Sep 11 11:32:07.393499 ip-10-0-135-183 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 11:32:07.393512 ip-10-0-135-183 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 11:32:07.393521 ip-10-0-135-183 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 11:32:07.393735 ip-10-0-135-183 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 11:32:17.618493 ip-10-0-135-183 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 11:32:17.618514 ip-10-0-135-183 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d6ff2a0b84cd4f7fb801da79a0d2dfa2 -- Sep 11 11:33:57.080071 ip-10-0-135-183 systemd[1]: Starting Kubernetes Kubelet... Sep 11 11:33:57.496452 ip-10-0-135-183 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 11:33:57.496452 ip-10-0-135-183 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 11:33:57.496452 ip-10-0-135-183 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 11:33:57.496452 ip-10-0-135-183 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 11:33:57.496452 ip-10-0-135-183 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 11:33:57.497477 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.497390 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 11:33:57.503570 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503555 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:33:57.503570 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503570 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503575 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503578 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503581 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503584 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503587 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503590 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503593 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503596 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503599 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503601 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503606 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503610 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503613 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503616 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503619 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503622 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503625 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503627 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:33:57.503635 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503630 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503632 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503635 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503637 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503645 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503648 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503651 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503654 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503657 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503660 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503662 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503664 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503667 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503669 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503672 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503674 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503677 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503679 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503681 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503684 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:33:57.504101 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503686 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503689 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503691 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503694 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503696 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503700 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503703 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503706 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503709 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503712 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503714 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503717 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503720 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503722 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503725 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503728 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503730 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503733 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503735 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:33:57.504579 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503737 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503741 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503744 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503746 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503748 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503751 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503753 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503756 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503758 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503760 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503762 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503765 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503770 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503773 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503776 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503778 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503781 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503783 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503796 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503799 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:33:57.505054 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503801 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503804 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503806 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503809 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503811 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503814 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503816 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503819 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503823 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503825 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503828 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503831 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503833 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503836 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503839 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503841 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503843 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503846 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503848 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.503851 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:33:57.505534 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504541 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504547 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504550 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504553 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504557 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504562 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504565 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504568 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504570 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504573 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504575 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504579 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504582 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504585 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504589 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504594 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504598 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504603 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504607 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:33:57.506037 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504633 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504669 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504674 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504677 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504680 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504684 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504687 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504690 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504692 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504695 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504697 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504700 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504702 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504705 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504707 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504710 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504712 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504715 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504717 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504720 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:33:57.506510 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504722 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504725 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504727 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504730 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504732 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504735 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504737 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504740 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504742 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504745 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504747 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504752 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504755 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504758 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504767 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504770 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504772 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504775 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504777 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504781 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:33:57.507010 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504783 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504799 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504802 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504805 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504807 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504811 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504815 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504819 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504821 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504824 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504831 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504834 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504836 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504839 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504841 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504844 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504846 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504849 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504851 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:33:57.507493 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504854 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504856 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504859 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504862 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504864 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504867 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504872 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504875 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504879 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504881 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504884 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504886 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504889 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504892 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504894 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504897 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504899 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504902 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504904 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:33:57.507966 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504907 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.504909 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505303 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505309 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505312 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505315 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505317 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505320 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505322 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505325 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505327 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505330 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505332 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505335 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505337 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505340 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505342 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505345 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505347 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:33:57.508444 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505353 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505357 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505361 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505364 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505367 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505369 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505372 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505375 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505378 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505381 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505383 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505386 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505388 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505391 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505393 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505396 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505399 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505402 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505404 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505407 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:33:57.508915 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505410 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505413 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505415 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505418 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505420 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505423 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505425 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505427 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505430 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505432 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505435 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505437 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505441 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505444 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505446 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505449 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505452 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505454 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505457 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:33:57.509477 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505459 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505461 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505464 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505467 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505469 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505472 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505474 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505478 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505481 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505484 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505488 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505491 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505493 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505496 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505498 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505501 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505503 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505505 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505508 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505510 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:33:57.509947 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505513 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505516 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505518 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505521 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505523 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505527 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505529 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505532 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505535 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505539 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505543 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505546 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505549 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505551 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505554 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505557 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505559 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505562 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505564 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505566 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:33:57.510422 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505569 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505571 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.505575 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505652 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505690 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505697 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505702 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505706 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505710 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505714 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505718 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505721 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505724 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505728 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505731 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505734 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505737 2580 flags.go:64] FLAG: --cgroup-root="" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505741 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505744 2580 flags.go:64] FLAG: --client-ca-file="" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505747 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505750 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505753 2580 flags.go:64] FLAG: --cluster-domain="" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505757 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505760 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 11:33:57.510954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505763 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505766 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505770 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505772 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505775 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505778 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505781 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505784 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505804 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505808 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505812 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505816 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505820 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505822 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505825 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505828 2580 flags.go:64] FLAG: --enable-server="true" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505831 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505836 2580 flags.go:64] FLAG: --event-burst="100" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505839 2580 flags.go:64] FLAG: --event-qps="50" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505842 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505844 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505847 2580 flags.go:64] FLAG: --eviction-hard="" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505851 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505854 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505857 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 11:33:57.511529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505861 2580 flags.go:64] FLAG: --eviction-soft="" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505864 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505866 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505869 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505872 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505875 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505879 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505881 2580 flags.go:64] FLAG: --feature-gates="" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505885 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505888 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505891 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505894 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505897 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505900 2580 flags.go:64] FLAG: --help="false" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505903 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505906 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505908 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505913 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505916 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505919 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505922 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505925 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505927 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 11:33:57.512133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505930 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505933 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505936 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505939 2580 flags.go:64] FLAG: --kube-reserved="" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505942 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505944 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505947 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505950 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505953 2580 flags.go:64] FLAG: --lock-file="" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505957 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505960 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505962 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505967 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505970 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505973 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505976 2580 flags.go:64] FLAG: --logging-format="text" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505979 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505983 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505986 2580 flags.go:64] FLAG: --manifest-url="" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505988 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505993 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.505996 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506000 2580 flags.go:64] FLAG: --max-pods="110" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506002 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506006 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 11:33:57.512702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506008 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506012 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506015 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506018 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506021 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506028 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506031 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506034 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506037 2580 flags.go:64] FLAG: --pod-cidr="" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506040 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506046 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506049 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506052 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506055 2580 flags.go:64] FLAG: --port="10250" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506057 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506060 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ed78b237a2045a50" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506064 2580 flags.go:64] FLAG: --qos-reserved="" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506068 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506070 2580 flags.go:64] FLAG: --register-node="true" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506073 2580 flags.go:64] FLAG: --register-with-taints="" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506077 2580 flags.go:64] FLAG: --registry-burst="10" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506080 2580 flags.go:64] FLAG: --registry-qps="5" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506082 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506085 2580 flags.go:64] FLAG: --reserved-memory="" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506090 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 11:33:57.513347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506093 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506095 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506098 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506101 2580 flags.go:64] FLAG: --runonce="false" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506104 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506107 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506109 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506112 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506115 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506118 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506122 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506124 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506127 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506130 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506132 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506135 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506138 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506141 2580 flags.go:64] FLAG: --system-cgroups="" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506143 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506149 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506152 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506154 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506158 2580 flags.go:64] FLAG: --tls-min-version="" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506161 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506165 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 11:33:57.514079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506168 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506171 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506173 2580 flags.go:64] FLAG: --v="2" Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506177 2580 flags.go:64] FLAG: --version="false" Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506181 2580 flags.go:64] FLAG: --vmodule="" Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506184 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.506188 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506299 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506302 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506305 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506308 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506311 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506313 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506316 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506318 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506321 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506324 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506326 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506329 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506331 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506334 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:33:57.514719 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506336 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506339 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506341 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506344 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506346 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506348 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506351 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506353 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506356 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506358 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506363 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506366 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506369 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506371 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506374 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506376 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506379 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506381 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506384 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506387 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:33:57.515245 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506390 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506392 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506395 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506397 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506399 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506402 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506405 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506407 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506409 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506412 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506414 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506418 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506422 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506424 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506427 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506430 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506432 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506435 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506437 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:33:57.515747 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506440 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506442 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506444 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506448 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506452 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506455 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506458 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506461 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506463 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506466 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506468 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506471 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506474 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506476 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506479 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506482 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506484 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506487 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506490 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506493 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:33:57.516244 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506495 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506498 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506500 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506502 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506505 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506508 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506510 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506512 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506515 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506518 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506520 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506523 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506526 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506529 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506531 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506535 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506537 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506544 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506546 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506549 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:33:57.516732 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506551 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506554 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506556 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506562 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506565 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.506568 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.507374 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.514737 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.514754 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.515666 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.515673 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515728 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515734 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515739 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:33:57.517249 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515742 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515745 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515747 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515750 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515753 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515755 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515758 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515761 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515764 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515766 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515769 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515772 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515774 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515777 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515779 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515782 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515784 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515801 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515803 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515806 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:33:57.517600 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515808 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515811 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515814 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515816 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515819 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515822 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515826 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515829 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515831 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515834 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515836 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515840 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515844 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515847 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515850 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515853 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515856 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515858 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515862 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:33:57.518158 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515866 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515869 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515872 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515875 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515877 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515880 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515882 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515885 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515887 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515889 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515892 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515894 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515897 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515899 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515901 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515904 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515907 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515910 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515912 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515915 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:33:57.518897 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515919 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515922 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515924 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515927 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515929 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515932 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515934 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515936 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515939 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515941 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515944 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515946 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515949 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515951 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515954 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515956 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515958 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515961 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515963 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515965 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:33:57.519435 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515968 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515970 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515973 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515975 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515978 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515980 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515983 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515985 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515989 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515992 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515995 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.515998 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516001 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516004 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516007 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516009 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:33:57.519934 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516012 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.516017 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516125 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516129 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516132 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516135 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516138 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516140 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516143 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516146 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516148 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516151 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516153 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516155 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516158 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 11:33:57.520316 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516160 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516163 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516165 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516168 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516170 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516173 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516175 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516177 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516180 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516183 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516186 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516188 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516191 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516194 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516196 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516200 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516202 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516205 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516207 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 11:33:57.520704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516210 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516213 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516215 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516218 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516220 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516222 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516225 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516227 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516230 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516232 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516235 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516237 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516239 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516242 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516244 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516247 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516250 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516253 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516255 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516258 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 11:33:57.521196 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516260 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516263 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516266 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516269 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516272 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516274 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516277 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516279 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516282 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516285 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516287 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516290 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516292 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516294 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516297 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516299 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516302 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516304 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516306 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516309 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 11:33:57.521683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516312 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516314 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516318 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516321 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516324 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516327 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516330 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516332 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516335 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516338 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516342 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516345 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516348 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516351 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516354 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516357 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516359 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516362 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516364 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 11:33:57.522235 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516367 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516369 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516372 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516375 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516378 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516380 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516382 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:57.516385 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.516390 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.516490 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 11:33:57.522680 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.521886 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 11:33:57.523039 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.522823 2580 server.go:1019] "Starting client certificate rotation" Sep 11 11:33:57.523039 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.522929 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 11:33:57.523579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.523568 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 11:33:57.525666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.525652 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 11:33:57.530629 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.530616 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 11:33:57.541126 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.541106 2580 log.go:25] "Validated CRI v1 runtime API" Sep 11 11:33:57.545227 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.545215 2580 log.go:25] "Validated CRI v1 image API" Sep 11 11:33:57.546149 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.546129 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 11:33:57.549843 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.549826 2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c81af741-af60-4889-aeed-f79d35f85616:/dev/nvme0n1p3 f6afd45a-50ba-4eea-bad8-bb19bb4edec6:/dev/nvme0n1p4] Sep 11 11:33:57.549902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.549844 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 11:33:57.554272 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.554162 2580 manager.go:216] Machine: {Timestamp:2026-09-11 11:33:57.553668583 +0000 UTC m=+0.367506988 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098761 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2360608e4cd4c03ab5f881877e5e46 SystemUUID:ec236060-8e4c-d4c0-3ab5-f881877e5e46 BootID:d6ff2a0b-84cd-4f7f-b801-da79a0d2dfa2 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5c:0a:09:92:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5c:0a:09:92:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:d2:ee:68:e1:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 11:33:57.554272 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.554264 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 11:33:57.554390 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.554331 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 11:33:57.558400 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.558325 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 11:33:57.558533 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.558403 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-183.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 11:33:57.559263 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.559253 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 11:33:57.559300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.559267 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 11:33:57.559300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.559283 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 11:33:57.560091 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.560081 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 11:33:57.561133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.561123 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 11:33:57.561447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.561431 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 11:33:57.564463 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564451 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 11 11:33:57.564572 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564563 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 11:33:57.564608 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564591 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 11 11:33:57.564608 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564600 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 11:33:57.564678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564610 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 11:33:57.564678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564619 2580 kubelet.go:398] "Adding apiserver pod source" Sep 11 11:33:57.564678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564628 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 11:33:57.564958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564945 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 11:33:57.565004 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.564967 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 11:33:57.568182 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.568157 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 11:33:57.570531 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.570515 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 11:33:57.570594 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.570547 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 11:33:57.573250 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573225 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 11:33:57.573250 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573251 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573257 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573262 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573271 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573276 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573290 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573295 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573301 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573306 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573315 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 11:33:57.573383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.573320 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 11:33:57.574263 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.574253 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 11:33:57.574263 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.574264 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 11:33:57.576158 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.576135 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 11:33:57.576235 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.576164 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 11:33:57.578220 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.578207 2580 server.go:1268] "Started kubelet" Sep 11 11:33:57.578334 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.578310 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 11:33:57.578397 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.578314 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 11:33:57.578397 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.578374 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 11:33:57.579102 ip-10-0-135-183 systemd[1]: Started Kubernetes Kubelet. Sep 11 11:33:57.579394 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.579380 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 11 11:33:57.580780 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.580660 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 11:33:57.583330 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.583304 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 11:33:57.586537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.586519 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-183.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 11:33:57.587590 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.586580 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-183.ec2.internal.18d441040a1b7cb3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-183.ec2.internal,UID:ip-10-0-135-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-183.ec2.internal,},FirstTimestamp:2026-09-11 11:33:57.578177715 +0000 UTC m=+0.392016181,LastTimestamp:2026-09-11 11:33:57.578177715 +0000 UTC m=+0.392016181,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-183.ec2.internal,}" Sep 11 11:33:57.588168 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.588152 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 11:33:57.588812 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.588781 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 11:33:57.589031 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.589012 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:57.589503 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589486 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 11:33:57.589503 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589504 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 11:33:57.589645 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589486 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 11:33:57.589645 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589624 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 11 11:33:57.589645 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589630 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 11 11:33:57.589767 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589716 2580 factory.go:153] Registering CRI-O factory Sep 11 11:33:57.589767 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589730 2580 factory.go:223] Registration of the crio container factory successfully Sep 11 11:33:57.589888 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589777 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 11:33:57.589888 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589808 2580 factory.go:55] Registering systemd factory Sep 11 11:33:57.589888 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589816 2580 factory.go:223] Registration of the systemd container factory successfully Sep 11 11:33:57.589888 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589833 2580 factory.go:103] Registering Raw factory Sep 11 11:33:57.589888 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.589846 2580 manager.go:1196] Started watching for new ooms in manager Sep 11 11:33:57.590190 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.590178 2580 manager.go:349] Starting recovery of all containers Sep 11 11:33:57.593102 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.593058 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-183.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 11:33:57.593188 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.593167 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 11:33:57.600607 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.600584 2580 manager.go:354] Recovery completed Sep 11 11:33:57.603012 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.602990 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 11 11:33:57.605965 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.605953 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:57.610411 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.610385 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:57.610462 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.610423 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:57.610462 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.610434 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:57.610947 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.610933 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 11:33:57.610947 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.610944 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:57.611035 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.610947 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 11:33:57.611035 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.610985 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 11 11:33:57.612105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.612093 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:57.612155 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.612116 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:57.612155 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.612130 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:57.612239 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.612228 2580 policy_none.go:49] "None policy: Start" Sep 11 11:33:57.612273 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.612246 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 11:33:57.612273 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.612260 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 11:33:57.612515 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.612430 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-183.ec2.internal.18d441040c0753b1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-183.ec2.internal,UID:ip-10-0-135-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-183.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-183.ec2.internal,},FirstTimestamp:2026-09-11 11:33:57.610410929 +0000 UTC m=+0.424249335,LastTimestamp:2026-09-11 11:33:57.610410929 +0000 UTC m=+0.424249335,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-183.ec2.internal,}" Sep 11 11:33:57.612648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.612541 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwczb" Sep 11 11:33:57.613338 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.613323 2580 policy_none.go:47] "Start" Sep 11 11:33:57.619982 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.619968 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwczb" Sep 11 11:33:57.621476 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.621386 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-183.ec2.internal.18d441040c079829 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-183.ec2.internal,UID:ip-10-0-135-183.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-183.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-183.ec2.internal,},FirstTimestamp:2026-09-11 11:33:57.610428457 +0000 UTC m=+0.424266863,LastTimestamp:2026-09-11 11:33:57.610428457 +0000 UTC m=+0.424266863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-183.ec2.internal,}" Sep 11 11:33:57.646381 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.646367 2580 manager.go:337] "Starting Device Plugin manager" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.646411 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.646421 2580 server.go:87] "Starting device plugin registration server" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.646718 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.646731 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.646850 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.646925 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.646935 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.647403 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 11:33:57.657582 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.647451 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:57.685390 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.685362 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 11:33:57.686706 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.686691 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 11:33:57.686760 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.686739 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 11:33:57.686834 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.686763 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 11:33:57.686834 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.686813 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 11:33:57.689290 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.689271 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:33:57.747320 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.747239 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:57.748088 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.748074 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:57.748139 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.748102 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:57.748139 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.748113 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:57.748139 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.748134 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.756763 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.756751 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.756824 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.756770 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-183.ec2.internal\": node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:57.784600 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.784579 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:57.787780 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.787757 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal"] Sep 11 11:33:57.787863 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.787853 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:57.788661 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.788647 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:57.788713 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.788681 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:57.788713 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.788697 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:57.789924 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.789911 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:57.790058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.790117 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790080 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:57.790650 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790633 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:57.790650 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790643 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:57.790769 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790660 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:57.790769 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790665 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:57.790769 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790674 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:57.790769 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.790678 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:57.792563 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.792549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.792609 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.792580 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:57.793258 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.793228 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:57.793326 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.793266 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:57.793326 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.793281 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:57.807327 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.807308 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-183.ec2.internal\" not found" node="ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.811040 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.811025 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-183.ec2.internal\" not found" node="ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.885405 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.885382 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:57.890708 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.890692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93f10d8d545a409e6cc084480a99f98b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-183.ec2.internal\" (UID: \"93f10d8d545a409e6cc084480a99f98b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.890771 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.890723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5d1a469bddd0a53fe6426e246444b36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal\" (UID: \"f5d1a469bddd0a53fe6426e246444b36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.890771 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.890742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5d1a469bddd0a53fe6426e246444b36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal\" (UID: \"f5d1a469bddd0a53fe6426e246444b36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.986123 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:57.986096 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:57.991471 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.991452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5d1a469bddd0a53fe6426e246444b36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal\" (UID: \"f5d1a469bddd0a53fe6426e246444b36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.991539 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.991485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5d1a469bddd0a53fe6426e246444b36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal\" (UID: \"f5d1a469bddd0a53fe6426e246444b36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.991539 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.991509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93f10d8d545a409e6cc084480a99f98b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-183.ec2.internal\" (UID: \"93f10d8d545a409e6cc084480a99f98b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.991637 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.991536 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5d1a469bddd0a53fe6426e246444b36-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal\" (UID: \"f5d1a469bddd0a53fe6426e246444b36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.991637 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.991557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5d1a469bddd0a53fe6426e246444b36-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal\" (UID: \"f5d1a469bddd0a53fe6426e246444b36\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:57.991637 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:57.991573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93f10d8d545a409e6cc084480a99f98b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-183.ec2.internal\" (UID: \"93f10d8d545a409e6cc084480a99f98b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" Sep 11 11:33:58.086972 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.086891 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.109435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.109402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:58.114027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.113780 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" Sep 11 11:33:58.187647 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.187618 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.288126 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.288094 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.388668 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.388600 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.489080 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.489047 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.522448 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.522428 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 11:33:58.522866 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.522568 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 11:33:58.549579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.549555 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:33:58.590094 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.590071 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.625180 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.625152 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 11:28:58 +0000 UTC" deadline="2028-03-18 08:06:58.490303907 +0000 UTC" Sep 11 11:33:58.625180 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.625178 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13292h32m59.865129046s" Sep 11 11:33:58.691053 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.691005 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.791522 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.791486 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.816821 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.816781 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 11:33:58.817864 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.817845 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientMemory" Sep 11 11:33:58.817947 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.817875 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 11:33:58.817947 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.817884 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeHasSufficientPID" Sep 11 11:33:58.856250 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:58.856210 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d1a469bddd0a53fe6426e246444b36.slice/crio-8e1363521743f58ad92776dfc4d91ee570044bf5241800dde3403940559e4398 WatchSource:0}: Error finding container 8e1363521743f58ad92776dfc4d91ee570044bf5241800dde3403940559e4398: Status 404 returned error can't find the container with id 8e1363521743f58ad92776dfc4d91ee570044bf5241800dde3403940559e4398 Sep 11 11:33:58.856415 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:33:58.856403 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f10d8d545a409e6cc084480a99f98b.slice/crio-d0ad6d92919ac3fc26e730cf70309acbad5243c30c515627ee9255472a135928 WatchSource:0}: Error finding container d0ad6d92919ac3fc26e730cf70309acbad5243c30c515627ee9255472a135928: Status 404 returned error can't find the container with id d0ad6d92919ac3fc26e730cf70309acbad5243c30c515627ee9255472a135928 Sep 11 11:33:58.861756 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.861736 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:33:58.892404 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:33:58.892380 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-183.ec2.internal\" not found" Sep 11 11:33:58.921290 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.921269 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:33:58.989975 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:58.989910 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" Sep 11 11:33:59.000175 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:59.000155 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 11:33:59.003100 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:59.003087 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" Sep 11 11:33:59.014776 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:33:59.014757 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 11:34:00.334837 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.334818 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:34:00.693967 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.693927 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 11:34:00.745144 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.745118 2580 apiserver.go:52] "Watching apiserver" Sep 11 11:34:00.754392 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.754368 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 11:34:00.754841 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.754819 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-wr9xg","openshift-image-registry/node-ca-vskzf","openshift-multus/multus-additional-cni-plugins-rqpd6","openshift-multus/network-metrics-daemon-s57lp","openshift-network-diagnostics/network-check-target-8pb6s","openshift-ovn-kubernetes/ovnkube-node-j94vg","kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh","openshift-cluster-node-tuning-operator/tuned-h5589","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal","openshift-multus/multus-ktnrm","openshift-network-operator/iptables-alerter-zxf89","kube-system/global-pull-secret-syncer-7nn5q","kube-system/konnectivity-agent-cbw5p"] Sep 11 11:34:00.757444 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.757421 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:00.759656 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.759424 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 11:34:00.759760 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.759724 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 11:34:00.759760 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.759748 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fxgb9\"" Sep 11 11:34:00.759963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.759915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.761597 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.761579 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 11:34:00.761922 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.761900 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l5hn4\"" Sep 11 11:34:00.762104 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.762091 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 11:34:00.762330 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.762314 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 11:34:00.762758 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.762743 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 11:34:00.762960 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.762931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.763079 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.763012 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:00.764057 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.764004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.765679 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.765659 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 11:34:00.765978 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.765935 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j2psn\"" Sep 11 11:34:00.766312 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.766220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.766312 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.766237 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 11:34:00.766426 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.766409 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 11:34:00.767791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.767773 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 11:34:00.768937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.768917 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 11:34:00.769253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.769237 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26djp\"" Sep 11 11:34:00.769435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.769420 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 11:34:00.769616 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.769601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:00.769684 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.769649 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:00.770920 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.770901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:00.771118 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.770966 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:00.773842 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.773824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.775787 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.775764 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 11:34:00.776013 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.775997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 11:34:00.776433 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.776412 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l6k5s\"" Sep 11 11:34:00.776529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.776511 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 11:34:00.776892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.776636 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 11:34:00.776892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.776694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.776892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.776748 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 11:34:00.777095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.776970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 11:34:00.778484 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.778463 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 11:34:00.778899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.778871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.778978 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.778910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:00.778978 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.778935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-k8s-cni-cncf-io\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.778978 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.778957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/349237ab-d760-40d3-88f0-0425ddad5af4-multus-daemon-config\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779107 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.778995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-etc-kubernetes\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779107 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-system-cni-dir\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.779107 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebef619d-9d2e-4a27-9b1c-300daad95756-kubelet-config\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.779107 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebef619d-9d2e-4a27-9b1c-300daad95756-dbus\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.779261 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.779261 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-cni-multus\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779261 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea77448-b67e-44e0-a935-9d6c03745028-host\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.779261 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-cnibin\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.779447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-cni-binary-copy\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.779447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp54\" (UniqueName: \"kubernetes.io/projected/1932cdb7-696f-48f4-8519-6e624ae80d71-kube-api-access-xjp54\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.779447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779318 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-cnibin\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-socket-dir-parent\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfl4n\" (UniqueName: \"kubernetes.io/projected/349237ab-d760-40d3-88f0-0425ddad5af4-kube-api-access-cfl4n\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f853954-d580-4808-ad19-480303016858-konnectivity-ca\") pod \"konnectivity-agent-cbw5p\" (UID: \"1f853954-d580-4808-ad19-480303016858\") " pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:00.779447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjc8t\" (UniqueName: \"kubernetes.io/projected/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-kube-api-access-gjc8t\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/349237ab-d760-40d3-88f0-0425ddad5af4-cni-binary-copy\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-kubelet\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-system-cni-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-netns\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-hostroot\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea77448-b67e-44e0-a935-9d6c03745028-serviceca\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.779744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.780143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.780143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-cni-bin\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.780143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-conf-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.780143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-multus-certs\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.780143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f853954-d580-4808-ad19-480303016858-agent-certs\") pod \"konnectivity-agent-cbw5p\" (UID: \"1f853954-d580-4808-ad19-480303016858\") " pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:00.780143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.779987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-cni-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.780143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.780001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z8hb\" (UniqueName: \"kubernetes.io/projected/4ea77448-b67e-44e0-a935-9d6c03745028-kube-api-access-2z8hb\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.780640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.780622 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 11:34:00.780901 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.780873 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxl55\"" Sep 11 11:34:00.782943 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.782925 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.783562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.783542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-np7v8\"" Sep 11 11:34:00.783901 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.783880 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 11:34:00.784253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.784231 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 11:34:00.784506 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.784491 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 11:34:00.784597 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.784582 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 11:34:00.786095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.786075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.787143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.787124 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:00.787795 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.787515 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f6tzd\"" Sep 11 11:34:00.790910 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.790892 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:00.792612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.792433 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 11:34:00.792612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.792450 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 11:34:00.792612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.792529 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 11:34:00.792764 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.792705 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zvf8j\"" Sep 11 11:34:00.800772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.800734 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d2gff" Sep 11 11:34:00.808770 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.808750 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d2gff" Sep 11 11:34:00.869688 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.869664 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 11:34:00.872081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.872018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" event={"ID":"93f10d8d545a409e6cc084480a99f98b","Type":"ContainerStarted","Data":"d0ad6d92919ac3fc26e730cf70309acbad5243c30c515627ee9255472a135928"} Sep 11 11:34:00.873834 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.873804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" event={"ID":"f5d1a469bddd0a53fe6426e246444b36","Type":"ContainerStarted","Data":"8e1363521743f58ad92776dfc4d91ee570044bf5241800dde3403940559e4398"} Sep 11 11:34:00.881100 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-tuned\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.881205 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.881205 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.881205 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-cni-bin\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.881205 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-multus-certs\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysconfig\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysctl-d\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-host-slash\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f853954-d580-4808-ad19-480303016858-agent-certs\") pod \"konnectivity-agent-cbw5p\" (UID: \"1f853954-d580-4808-ad19-480303016858\") " pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdw9x\" (UniqueName: \"kubernetes.io/projected/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-kube-api-access-pdw9x\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-var-lib-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovnkube-script-lib\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.881404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-k8s-cni-cncf-io\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/349237ab-d760-40d3-88f0-0425ddad5af4-multus-daemon-config\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-etc-kubernetes\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-run\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgbd\" (UniqueName: \"kubernetes.io/projected/3d218a51-95f0-4a60-9ce3-7f046c0615df-kube-api-access-ckgbd\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881528 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-registration-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-var-lib-kubelet\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881574 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq585\" (UniqueName: \"kubernetes.io/projected/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-kube-api-access-tq585\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-device-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebef619d-9d2e-4a27-9b1c-300daad95756-kubelet-config\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebef619d-9d2e-4a27-9b1c-300daad95756-dbus\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-cni-multus\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea77448-b67e-44e0-a935-9d6c03745028-host\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-modprobe-d\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.881774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-systemd-units\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-systemd\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-host\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-cnibin\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp54\" (UniqueName: \"kubernetes.io/projected/1932cdb7-696f-48f4-8519-6e624ae80d71-kube-api-access-xjp54\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-socket-dir-parent\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfl4n\" (UniqueName: \"kubernetes.io/projected/349237ab-d760-40d3-88f0-0425ddad5af4-kube-api-access-cfl4n\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543efb58-4920-4bba-bb4b-a3a4411fa7a4-tmp-dir\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.881987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysctl-conf\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-sys\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjc8t\" (UniqueName: \"kubernetes.io/projected/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-kube-api-access-gjc8t\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/349237ab-d760-40d3-88f0-0425ddad5af4-cni-binary-copy\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/543efb58-4920-4bba-bb4b-a3a4411fa7a4-hosts-file\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-slash\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-run-netns\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882193 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-env-overrides\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-socket-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.882562 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-system-cni-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-hostroot\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea77448-b67e-44e0-a935-9d6c03745028-serviceca\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9st\" (UniqueName: \"kubernetes.io/projected/543efb58-4920-4bba-bb4b-a3a4411fa7a4-kube-api-access-8l9st\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-ovn\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-log-socket\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-cni-bin\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovn-node-metrics-cert\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-conf-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-kubelet\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-cni-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z8hb\" (UniqueName: \"kubernetes.io/projected/4ea77448-b67e-44e0-a935-9d6c03745028-kube-api-access-2z8hb\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-kubernetes\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-systemd\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s645h\" (UniqueName: \"kubernetes.io/projected/163cde0a-0c8d-4169-92db-3024d33611bf-kube-api-access-s645h\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-lib-modules\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.883439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-sys-fs\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-system-cni-dir\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-etc-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882841 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d218a51-95f0-4a60-9ce3-7f046c0615df-tmp\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-cni-binary-copy\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-cnibin\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-node-log\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovnkube-config\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.882985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f853954-d580-4808-ad19-480303016858-konnectivity-ca\") pod \"konnectivity-agent-cbw5p\" (UID: \"1f853954-d580-4808-ad19-480303016858\") " pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-kubelet\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-iptables-alerter-script\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.884297 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-cni-netd\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883189 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-netns\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-netns\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.883388 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.883455 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret podName:ebef619d-9d2e-4a27-9b1c-300daad95756 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:01.383428577 +0000 UTC m=+3.018021705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret") pod "global-pull-secret-syncer-7nn5q" (UID: "ebef619d-9d2e-4a27-9b1c-300daad95756") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883820 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-cni-bin\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.883866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-multus-certs\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.884159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ea77448-b67e-44e0-a935-9d6c03745028-host\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.884210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-cnibin\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.884294 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.884357 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs podName:6df5f2cf-051f-4ab9-a41d-7f0305cc486f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:01.384338895 +0000 UTC m=+3.018932021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs") pod "network-metrics-daemon-s57lp" (UID: "6df5f2cf-051f-4ab9-a41d-7f0305cc486f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.884405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-run-k8s-cni-cncf-io\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.884421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-system-cni-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.884498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-hostroot\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.884543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1932cdb7-696f-48f4-8519-6e624ae80d71-system-cni-dir\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.885089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.884631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-cnibin\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885053 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/349237ab-d760-40d3-88f0-0425ddad5af4-multus-daemon-config\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ea77448-b67e-44e0-a935-9d6c03745028-serviceca\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-etc-kubernetes\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ebef619d-9d2e-4a27-9b1c-300daad95756-kubelet-config\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ebef619d-9d2e-4a27-9b1c-300daad95756-dbus\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f853954-d580-4808-ad19-480303016858-konnectivity-ca\") pod \"konnectivity-agent-cbw5p\" (UID: \"1f853954-d580-4808-ad19-480303016858\") " pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-kubelet\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-socket-dir-parent\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885622 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-host-var-lib-cni-multus\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-cni-binary-copy\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.885869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-cni-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.886475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.885949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/349237ab-d760-40d3-88f0-0425ddad5af4-multus-conf-dir\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.886475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.886093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/349237ab-d760-40d3-88f0-0425ddad5af4-cni-binary-copy\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.886475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.886245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1932cdb7-696f-48f4-8519-6e624ae80d71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.886748 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.886718 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 11:34:00.891666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.891501 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f853954-d580-4808-ad19-480303016858-agent-certs\") pod \"konnectivity-agent-cbw5p\" (UID: \"1f853954-d580-4808-ad19-480303016858\") " pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:00.894175 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.894155 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:34:00.894268 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.894179 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:34:00.894268 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.894191 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ln89l for pod openshift-network-diagnostics/network-check-target-8pb6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:00.894268 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:00.894248 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l podName:3d689159-7ed7-4c57-af29-1b64ac0491b6 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:01.394232318 +0000 UTC m=+3.028825457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ln89l" (UniqueName: "kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l") pod "network-check-target-8pb6s" (UID: "3d689159-7ed7-4c57-af29-1b64ac0491b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:00.896183 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.896159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfl4n\" (UniqueName: \"kubernetes.io/projected/349237ab-d760-40d3-88f0-0425ddad5af4-kube-api-access-cfl4n\") pod \"multus-ktnrm\" (UID: \"349237ab-d760-40d3-88f0-0425ddad5af4\") " pod="openshift-multus/multus-ktnrm" Sep 11 11:34:00.897392 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.897352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z8hb\" (UniqueName: \"kubernetes.io/projected/4ea77448-b67e-44e0-a935-9d6c03745028-kube-api-access-2z8hb\") pod \"node-ca-vskzf\" (UID: \"4ea77448-b67e-44e0-a935-9d6c03745028\") " pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:00.897482 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.897402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp54\" (UniqueName: \"kubernetes.io/projected/1932cdb7-696f-48f4-8519-6e624ae80d71-kube-api-access-xjp54\") pod \"multus-additional-cni-plugins-rqpd6\" (UID: \"1932cdb7-696f-48f4-8519-6e624ae80d71\") " pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:00.898514 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.898493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjc8t\" (UniqueName: \"kubernetes.io/projected/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-kube-api-access-gjc8t\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:00.984056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.983962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-kubernetes\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.983992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-systemd\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s645h\" (UniqueName: \"kubernetes.io/projected/163cde0a-0c8d-4169-92db-3024d33611bf-kube-api-access-s645h\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.984056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-lib-modules\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-sys-fs\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-kubernetes\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-etc-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-systemd\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d218a51-95f0-4a60-9ce3-7f046c0615df-tmp\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-node-log\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovnkube-config\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984305 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-iptables-alerter-script\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-cni-netd\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984355 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-tuned\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysconfig\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysctl-d\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-host-slash\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdw9x\" (UniqueName: \"kubernetes.io/projected/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-kube-api-access-pdw9x\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-var-lib-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovnkube-script-lib\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-run\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgbd\" (UniqueName: \"kubernetes.io/projected/3d218a51-95f0-4a60-9ce3-7f046c0615df-kube-api-access-ckgbd\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-registration-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-cni-netd\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-var-lib-kubelet\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.984791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq585\" (UniqueName: \"kubernetes.io/projected/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-kube-api-access-tq585\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysctl-d\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-device-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984768 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-sys-fs\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-modprobe-d\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984802 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-lib-modules\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-node-log\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-systemd-units\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-var-lib-kubelet\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985383 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-device-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985461 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovnkube-config\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-modprobe-d\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-registration-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysconfig\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-etc-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.985765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-iptables-alerter-script\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.984804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-systemd-units\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-systemd\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985685 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-run\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-host\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543efb58-4920-4bba-bb4b-a3a4411fa7a4-tmp-dir\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-host-slash\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysctl-conf\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-sys\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-var-lib-openvswitch\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/543efb58-4920-4bba-bb4b-a3a4411fa7a4-hosts-file\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-systemd\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-slash\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-run-netns\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-slash\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-sys\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-host\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.986558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-env-overrides\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-socket-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovnkube-script-lib\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.985997 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9st\" (UniqueName: \"kubernetes.io/projected/543efb58-4920-4bba-bb4b-a3a4411fa7a4-kube-api-access-8l9st\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-ovn\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-log-socket\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-cni-bin\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovn-node-metrics-cert\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-kubelet\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-sysctl-conf\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-cni-bin\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-run-ovn\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-run-netns\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/543efb58-4920-4bba-bb4b-a3a4411fa7a4-hosts-file\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986432 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/543efb58-4920-4bba-bb4b-a3a4411fa7a4-tmp-dir\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-host-kubelet\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-env-overrides\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-log-socket\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.987431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.986752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/163cde0a-0c8d-4169-92db-3024d33611bf-socket-dir\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.988098 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.987700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d218a51-95f0-4a60-9ce3-7f046c0615df-etc-tuned\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.988098 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.987897 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d218a51-95f0-4a60-9ce3-7f046c0615df-tmp\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.988780 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.988732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-ovn-node-metrics-cert\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.994242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.994214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgbd\" (UniqueName: \"kubernetes.io/projected/3d218a51-95f0-4a60-9ce3-7f046c0615df-kube-api-access-ckgbd\") pod \"tuned-h5589\" (UID: \"3d218a51-95f0-4a60-9ce3-7f046c0615df\") " pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:00.994865 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.994836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9st\" (UniqueName: \"kubernetes.io/projected/543efb58-4920-4bba-bb4b-a3a4411fa7a4-kube-api-access-8l9st\") pod \"node-resolver-wr9xg\" (UID: \"543efb58-4920-4bba-bb4b-a3a4411fa7a4\") " pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:00.995354 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.995335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s645h\" (UniqueName: \"kubernetes.io/projected/163cde0a-0c8d-4169-92db-3024d33611bf-kube-api-access-s645h\") pod \"aws-ebs-csi-driver-node-lnhlh\" (UID: \"163cde0a-0c8d-4169-92db-3024d33611bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:00.995927 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.995910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq585\" (UniqueName: \"kubernetes.io/projected/2c1cf753-cfd4-4130-b5df-82d3c12f2fbd-kube-api-access-tq585\") pod \"ovnkube-node-j94vg\" (UID: \"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:00.996273 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:00.996256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdw9x\" (UniqueName: \"kubernetes.io/projected/6ac1cee7-b05b-4e9c-9648-e40bc29f341a-kube-api-access-pdw9x\") pod \"iptables-alerter-zxf89\" (UID: \"6ac1cee7-b05b-4e9c-9648-e40bc29f341a\") " pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:01.071273 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.071237 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:01.081105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.081080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ktnrm" Sep 11 11:34:01.094997 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.094970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vskzf" Sep 11 11:34:01.103687 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.103667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" Sep 11 11:34:01.112471 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.112452 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:01.119108 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.119091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr9xg" Sep 11 11:34:01.128754 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.128736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" Sep 11 11:34:01.136380 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.136360 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h5589" Sep 11 11:34:01.143899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.143882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxf89" Sep 11 11:34:01.389119 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.389055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:01.389119 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.389095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:01.389310 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.389197 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:01.389310 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.389203 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:01.389310 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.389249 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs podName:6df5f2cf-051f-4ab9-a41d-7f0305cc486f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:02.389234551 +0000 UTC m=+4.023827662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs") pod "network-metrics-daemon-s57lp" (UID: "6df5f2cf-051f-4ab9-a41d-7f0305cc486f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:01.389310 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.389266 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret podName:ebef619d-9d2e-4a27-9b1c-300daad95756 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:02.389256669 +0000 UTC m=+4.023849785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret") pod "global-pull-secret-syncer-7nn5q" (UID: "ebef619d-9d2e-4a27-9b1c-300daad95756") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:01.490197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.490149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:01.490354 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.490312 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:34:01.490354 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.490334 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:34:01.490354 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.490346 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ln89l for pod openshift-network-diagnostics/network-check-target-8pb6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:01.490487 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.490412 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l podName:3d689159-7ed7-4c57-af29-1b64ac0491b6 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:02.490391734 +0000 UTC m=+4.124984871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ln89l" (UniqueName: "kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l") pod "network-check-target-8pb6s" (UID: "3d689159-7ed7-4c57-af29-1b64ac0491b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:01.812238 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.812157 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 11:29:00 +0000 UTC" deadline="2028-04-24 19:43:37.579641499 +0000 UTC" Sep 11 11:34:01.812238 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.812192 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14192h9m35.767453662s" Sep 11 11:34:01.866853 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.866818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:01.867014 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.866938 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:01.867334 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:01.867306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:01.867456 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:01.867405 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:02.088028 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:02.088002 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea77448_b67e_44e0_a935_9d6c03745028.slice/crio-930d1becd295290d7f8e34c2cf6d22343c18ac0eb9d28edda7daca5f08688d59 WatchSource:0}: Error finding container 930d1becd295290d7f8e34c2cf6d22343c18ac0eb9d28edda7daca5f08688d59: Status 404 returned error can't find the container with id 930d1becd295290d7f8e34c2cf6d22343c18ac0eb9d28edda7daca5f08688d59 Sep 11 11:34:02.089623 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:02.089593 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d218a51_95f0_4a60_9ce3_7f046c0615df.slice/crio-d1ba20498ec23d0bf8b46462dddf8bd4585b6c3252c33137c4ecde572c1e62a6 WatchSource:0}: Error finding container d1ba20498ec23d0bf8b46462dddf8bd4585b6c3252c33137c4ecde572c1e62a6: Status 404 returned error can't find the container with id d1ba20498ec23d0bf8b46462dddf8bd4585b6c3252c33137c4ecde572c1e62a6 Sep 11 11:34:02.395590 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.395556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:02.395725 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.395617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:02.395763 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.395722 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:02.395763 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.395754 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:02.395865 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.395793 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret podName:ebef619d-9d2e-4a27-9b1c-300daad95756 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:04.395774824 +0000 UTC m=+6.030367939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret") pod "global-pull-secret-syncer-7nn5q" (UID: "ebef619d-9d2e-4a27-9b1c-300daad95756") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:02.395865 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.395808 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs podName:6df5f2cf-051f-4ab9-a41d-7f0305cc486f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:04.395802138 +0000 UTC m=+6.030395250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs") pod "network-metrics-daemon-s57lp" (UID: "6df5f2cf-051f-4ab9-a41d-7f0305cc486f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:02.496968 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.496935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:02.497141 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.497101 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:34:02.497141 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.497119 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:34:02.497141 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.497137 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ln89l for pod openshift-network-diagnostics/network-check-target-8pb6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:02.497239 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.497191 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l podName:3d689159-7ed7-4c57-af29-1b64ac0491b6 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:04.497174366 +0000 UTC m=+6.131767494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ln89l" (UniqueName: "kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l") pod "network-check-target-8pb6s" (UID: "3d689159-7ed7-4c57-af29-1b64ac0491b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:02.812881 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.812788 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 11:29:00 +0000 UTC" deadline="2028-06-28 05:44:45.774290322 +0000 UTC" Sep 11 11:34:02.812881 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.812835 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15738h10m42.961460434s" Sep 11 11:34:02.868925 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.868895 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:02.869098 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:02.869027 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:02.893404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.893363 2580 generic.go:356] "Generic (PLEG): container finished" podID="f5d1a469bddd0a53fe6426e246444b36" containerID="ec6b6704570e41d7dc799b7099c2f35695b5e7ebfc68b14dfd85533dfa7b524c" exitCode=0 Sep 11 11:34:02.893677 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.893645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" event={"ID":"f5d1a469bddd0a53fe6426e246444b36","Type":"ContainerDied","Data":"ec6b6704570e41d7dc799b7099c2f35695b5e7ebfc68b14dfd85533dfa7b524c"} Sep 11 11:34:02.899307 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.899254 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ktnrm" event={"ID":"349237ab-d760-40d3-88f0-0425ddad5af4","Type":"ContainerStarted","Data":"0f957565bdebe634fe94907ad1ea7f2d83365a6047b2d50f6532df8b2335f89b"} Sep 11 11:34:02.925095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.924988 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" event={"ID":"163cde0a-0c8d-4169-92db-3024d33611bf","Type":"ContainerStarted","Data":"52c73ed13dcb203c45a944c798f39cfc0ecc9e0ae34ff3e1821ee7d895dd9223"} Sep 11 11:34:02.937105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.937022 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cbw5p" event={"ID":"1f853954-d580-4808-ad19-480303016858","Type":"ContainerStarted","Data":"3be072ceab01d641af4469aca180ab55e80afef5453aa85573ac1aaf4cca6134"} Sep 11 11:34:02.961264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.961227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr9xg" event={"ID":"543efb58-4920-4bba-bb4b-a3a4411fa7a4","Type":"ContainerStarted","Data":"133da36b6dff107f230bcf432522652caba0fed8268a3490661984d091f199fa"} Sep 11 11:34:02.971325 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.971287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vskzf" event={"ID":"4ea77448-b67e-44e0-a935-9d6c03745028","Type":"ContainerStarted","Data":"930d1becd295290d7f8e34c2cf6d22343c18ac0eb9d28edda7daca5f08688d59"} Sep 11 11:34:02.992401 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:02.992268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"c870b1a7ada1b0b59fd232f5b69f430748f1e382ddc6b46f600be4dc0e548d0f"} Sep 11 11:34:03.017192 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:03.017109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxf89" event={"ID":"6ac1cee7-b05b-4e9c-9648-e40bc29f341a","Type":"ContainerStarted","Data":"31b98991a9108380e98b44db5cade296d81d4a8515b077773c84991a214a6801"} Sep 11 11:34:03.032930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:03.032890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerStarted","Data":"e9fb68130b2f2626e6d19d50c9104aaa6046032c9feff133a01b97064a0f829a"} Sep 11 11:34:03.043618 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:03.043574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h5589" event={"ID":"3d218a51-95f0-4a60-9ce3-7f046c0615df","Type":"ContainerStarted","Data":"d1ba20498ec23d0bf8b46462dddf8bd4585b6c3252c33137c4ecde572c1e62a6"} Sep 11 11:34:03.052566 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:03.051757 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" event={"ID":"93f10d8d545a409e6cc084480a99f98b","Type":"ContainerStarted","Data":"5902ad61acff4ddba7df6c08626380073558b0672eb27f33701e23d497aa6f03"} Sep 11 11:34:03.866956 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:03.866703 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:03.867443 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:03.867100 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:03.867524 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:03.867507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:03.867637 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:03.867612 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:04.079492 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:04.079342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" event={"ID":"f5d1a469bddd0a53fe6426e246444b36","Type":"ContainerStarted","Data":"fd7e9e25f865073890ecc859725fc71169536409e17c01308f9ead709da3f3d5"} Sep 11 11:34:04.095521 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:04.095453 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-183.ec2.internal" podStartSLOduration=4.095436137 podStartE2EDuration="4.095436137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:03.067004925 +0000 UTC m=+4.701598060" watchObservedRunningTime="2026-09-11 11:34:04.095436137 +0000 UTC m=+5.730029268" Sep 11 11:34:04.095736 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:04.095713 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-183.ec2.internal" podStartSLOduration=4.095706231 podStartE2EDuration="4.095706231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:04.095249306 +0000 UTC m=+5.729842438" watchObservedRunningTime="2026-09-11 11:34:04.095706231 +0000 UTC m=+5.730299363" Sep 11 11:34:04.413765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:04.413718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:04.413948 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:04.413774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:04.413948 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.413910 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:04.414098 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.413982 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs podName:6df5f2cf-051f-4ab9-a41d-7f0305cc486f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:08.413962511 +0000 UTC m=+10.048555638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs") pod "network-metrics-daemon-s57lp" (UID: "6df5f2cf-051f-4ab9-a41d-7f0305cc486f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:04.414415 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.414387 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:04.414505 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.414441 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret podName:ebef619d-9d2e-4a27-9b1c-300daad95756 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:08.41442687 +0000 UTC m=+10.049019988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret") pod "global-pull-secret-syncer-7nn5q" (UID: "ebef619d-9d2e-4a27-9b1c-300daad95756") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:04.515392 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:04.514745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:04.515392 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.514938 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:34:04.515392 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.514960 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:34:04.515392 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.514975 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ln89l for pod openshift-network-diagnostics/network-check-target-8pb6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:04.515392 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.515057 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l podName:3d689159-7ed7-4c57-af29-1b64ac0491b6 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:08.515017453 +0000 UTC m=+10.149610565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ln89l" (UniqueName: "kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l") pod "network-check-target-8pb6s" (UID: "3d689159-7ed7-4c57-af29-1b64ac0491b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:04.868895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:04.868797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:04.869440 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:04.868926 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:05.867081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:05.866426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:05.867081 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:05.866551 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:05.867081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:05.866923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:05.867081 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:05.867021 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:06.866565 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:06.866532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:06.866994 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:06.866671 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:07.866650 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:07.866614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:07.867125 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:07.866753 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:07.867125 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:07.866801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:07.867125 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:07.866887 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:08.449480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:08.449389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:08.449480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:08.449446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:08.449728 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.449547 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:08.449728 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.449557 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:08.449728 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.449610 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs podName:6df5f2cf-051f-4ab9-a41d-7f0305cc486f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:16.449595947 +0000 UTC m=+18.084189061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs") pod "network-metrics-daemon-s57lp" (UID: "6df5f2cf-051f-4ab9-a41d-7f0305cc486f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:08.449728 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.449640 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret podName:ebef619d-9d2e-4a27-9b1c-300daad95756 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:16.449618353 +0000 UTC m=+18.084211465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret") pod "global-pull-secret-syncer-7nn5q" (UID: "ebef619d-9d2e-4a27-9b1c-300daad95756") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:08.550322 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:08.550290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:08.550496 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.550471 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:34:08.550496 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.550494 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:34:08.550602 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.550507 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ln89l for pod openshift-network-diagnostics/network-check-target-8pb6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:08.550602 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.550562 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l podName:3d689159-7ed7-4c57-af29-1b64ac0491b6 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:16.550544269 +0000 UTC m=+18.185137382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ln89l" (UniqueName: "kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l") pod "network-check-target-8pb6s" (UID: "3d689159-7ed7-4c57-af29-1b64ac0491b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:08.867887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:08.867788 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:08.868302 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:08.867890 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:09.866640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:09.866603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:09.866831 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:09.866618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:09.866831 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:09.866755 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:09.866831 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:09.866802 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:10.868093 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:10.868058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:10.868466 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:10.868190 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:11.866497 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:11.866444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:11.866661 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:11.866574 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:11.866661 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:11.866633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:11.866781 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:11.866755 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:12.866354 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:12.866271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:12.866832 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:12.866409 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:13.866308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:13.866272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:13.866486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:13.866272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:13.866486 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:13.866418 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:13.866486 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:13.866460 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:14.867129 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:14.867098 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:14.867568 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:14.867223 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:15.867201 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:15.867172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:15.867201 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:15.867201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:15.867687 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:15.867292 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:15.867687 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:15.867425 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:16.510430 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:16.510386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:16.510697 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:16.510443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:16.510697 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.510532 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:16.510697 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.510543 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:16.511003 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.510975 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret podName:ebef619d-9d2e-4a27-9b1c-300daad95756 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:32.510589075 +0000 UTC m=+34.145182198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret") pod "global-pull-secret-syncer-7nn5q" (UID: "ebef619d-9d2e-4a27-9b1c-300daad95756") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:16.511185 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.511016 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs podName:6df5f2cf-051f-4ab9-a41d-7f0305cc486f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:32.51099704 +0000 UTC m=+34.145590151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs") pod "network-metrics-daemon-s57lp" (UID: "6df5f2cf-051f-4ab9-a41d-7f0305cc486f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:16.611623 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:16.611592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:16.611824 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.611802 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:34:16.611891 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.611832 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:34:16.611891 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.611846 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ln89l for pod openshift-network-diagnostics/network-check-target-8pb6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:16.611983 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.611914 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l podName:3d689159-7ed7-4c57-af29-1b64ac0491b6 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:32.611894018 +0000 UTC m=+34.246487146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ln89l" (UniqueName: "kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l") pod "network-check-target-8pb6s" (UID: "3d689159-7ed7-4c57-af29-1b64ac0491b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:16.867321 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:16.867244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:16.867754 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:16.867369 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:17.866230 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:17.866202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:17.866412 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:17.866331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:17.866412 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:17.866390 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:17.866522 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:17.866475 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:18.866968 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:18.866938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:18.867436 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:18.867047 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:19.867527 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:19.867129 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:19.867527 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:19.867253 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:19.867527 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:19.867338 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:19.867527 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:19.867482 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:20.130972 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.130609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ktnrm" event={"ID":"349237ab-d760-40d3-88f0-0425ddad5af4","Type":"ContainerStarted","Data":"95f1ff26dfa3d8fcfae230340c1d9bfc1918bf1deb0a3a620a120c983e171e55"} Sep 11 11:34:20.132329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.132297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" event={"ID":"163cde0a-0c8d-4169-92db-3024d33611bf","Type":"ContainerStarted","Data":"ac62af223011bbb7ca1788ebb509ff8fc4b5ca70bf3c6fc173b307f82475e5c7"} Sep 11 11:34:20.133932 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.133909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cbw5p" event={"ID":"1f853954-d580-4808-ad19-480303016858","Type":"ContainerStarted","Data":"fc5c2f7e8ff553dceadd91598b3db34c73f5eed56f03809370e003cc19e56435"} Sep 11 11:34:20.135666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.135478 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr9xg" event={"ID":"543efb58-4920-4bba-bb4b-a3a4411fa7a4","Type":"ContainerStarted","Data":"7a06347c01796eecd50185cf20d656534b561c3e1a5069af83a4ad4dfe105788"} Sep 11 11:34:20.137143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.137119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vskzf" event={"ID":"4ea77448-b67e-44e0-a935-9d6c03745028","Type":"ContainerStarted","Data":"a23a9559fb138af5a60815ba3b20fcf3cc57499d89fb3fe6a182ec16152b0438"} Sep 11 11:34:20.139757 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.139730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"97b55cc54d53912b28e87eabb0ad55fb80af05ea93e13f1bed69a0a4abab0e15"} Sep 11 11:34:20.139848 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.139790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"6b28836157bf6d644e825c3a4f339cc0bca817db512d7e00b25fbff4921e9c3a"} Sep 11 11:34:20.139848 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.139826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"86eb6b950ba55424ab7198b6dc1213685631bfbbfa41438e0363764a358960c7"} Sep 11 11:34:20.141629 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.141589 2580 generic.go:356] "Generic (PLEG): container finished" podID="1932cdb7-696f-48f4-8519-6e624ae80d71" containerID="b5430bd6e013fddfc3fad1025b6f4f65dcdf24f79f50ae176cf552690df554cb" exitCode=0 Sep 11 11:34:20.141741 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.141662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerDied","Data":"b5430bd6e013fddfc3fad1025b6f4f65dcdf24f79f50ae176cf552690df554cb"} Sep 11 11:34:20.143732 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.143697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h5589" event={"ID":"3d218a51-95f0-4a60-9ce3-7f046c0615df","Type":"ContainerStarted","Data":"74c16850861bcc2dedd7a19d7c0a5988dc51528498c17c5128ff2aaf47ea4828"} Sep 11 11:34:20.180085 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.180020 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ktnrm" podStartSLOduration=4.452823985 podStartE2EDuration="22.180004533s" totalImagesPullingTime="17.727180548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:20.179707401 +0000 UTC m=+21.814300542" watchObservedRunningTime="2026-09-11 11:34:20.180004533 +0000 UTC m=+21.814597666" Sep 11 11:34:20.228203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.228155 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wr9xg" podStartSLOduration=3.5577928290000003 podStartE2EDuration="21.228140054s" totalImagesPullingTime="17.670347225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:20.228133883 +0000 UTC m=+21.862727018" watchObservedRunningTime="2026-09-11 11:34:20.228140054 +0000 UTC m=+21.862733186" Sep 11 11:34:20.245807 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.245771 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h5589" podStartSLOduration=3.571883488 podStartE2EDuration="21.245757368s" totalImagesPullingTime="17.67387388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:20.245252214 +0000 UTC m=+21.879845346" watchObservedRunningTime="2026-09-11 11:34:20.245757368 +0000 UTC m=+21.880350500" Sep 11 11:34:20.261157 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.261113 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cbw5p" podStartSLOduration=8.64809199 podStartE2EDuration="22.261096381s" totalImagesPullingTime="13.613004391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:20.260707847 +0000 UTC m=+21.895300983" watchObservedRunningTime="2026-09-11 11:34:20.261096381 +0000 UTC m=+21.895689515" Sep 11 11:34:20.288103 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.288069 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vskzf" podStartSLOduration=3.616120742 podStartE2EDuration="21.288060436s" totalImagesPullingTime="17.671939694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:20.287684721 +0000 UTC m=+21.922277865" watchObservedRunningTime="2026-09-11 11:34:20.288060436 +0000 UTC m=+21.922653568" Sep 11 11:34:20.866892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:20.866683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:20.867077 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:20.866979 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:21.148986 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.148937 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"831797e73a0f95ba5b5f3c1703ac66c6ca803cc81223186baaf90e397db73cdc"} Sep 11 11:34:21.149772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.149063 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"7f1fece1fe9395f5b05a4828988ead55a845fba2a4a7fa203ca00852f5709c66"} Sep 11 11:34:21.149772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.149112 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"08566f06f6c40fd6238ab6aaf4aa004f96e16007a072bb3897bfa1248fb1fd1e"} Sep 11 11:34:21.151084 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.150650 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxf89" event={"ID":"6ac1cee7-b05b-4e9c-9648-e40bc29f341a","Type":"ContainerStarted","Data":"15337b3de82e611b61677b6e98dc38a270c755fd2ab161882146ebd5345e9d05"} Sep 11 11:34:21.574937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.574916 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 11:34:21.837700 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.837617 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:21.837700 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.837442 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T11:34:21.574933876Z","UUID":"75fc2724-d8cb-46fa-b040-b45f7779acd1","Handler":null,"Name":"","Endpoint":""} Sep 11 11:34:21.838174 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.838152 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:21.839093 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.839075 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 11:34:21.839184 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.839108 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 11:34:21.863862 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.863806 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zxf89" podStartSLOduration=5.204363784 podStartE2EDuration="22.86378733s" totalImagesPullingTime="17.659423546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:21.166459189 +0000 UTC m=+22.801052322" watchObservedRunningTime="2026-09-11 11:34:21.86378733 +0000 UTC m=+23.498380465" Sep 11 11:34:21.866648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.866625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:21.866766 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:21.866729 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:21.866827 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:21.866785 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:21.866903 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:21.866883 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:22.155660 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:22.155564 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" event={"ID":"163cde0a-0c8d-4169-92db-3024d33611bf","Type":"ContainerStarted","Data":"e5d9f98a66f217439a2f3ec133f0d8770e35efe0a089f69e71ff4dd580827bea"} Sep 11 11:34:22.156115 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:22.155871 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:22.156317 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:22.156303 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cbw5p" Sep 11 11:34:22.867167 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:22.867137 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:22.867303 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:22.867265 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:23.160073 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:23.159970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" event={"ID":"163cde0a-0c8d-4169-92db-3024d33611bf","Type":"ContainerStarted","Data":"2cb787aaf040671654539c009b66a5629fe63c47167bcc0eeef7cfd3373b5f1f"} Sep 11 11:34:23.163543 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:23.163521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"e9d85cf04f8134656e56065fbff0125eb22557cbfcaee536aab3d23385b9a6d2"} Sep 11 11:34:23.177970 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:23.177932 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnhlh" podStartSLOduration=3.817647521 podStartE2EDuration="24.177920423s" totalImagesPullingTime="20.360272902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:23.177611317 +0000 UTC m=+24.812204461" watchObservedRunningTime="2026-09-11 11:34:23.177920423 +0000 UTC m=+24.812513552" Sep 11 11:34:23.866678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:23.866646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:23.866881 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:23.866651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:23.866881 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:23.866777 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:23.866881 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:23.866862 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:24.866682 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:24.866641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:24.867107 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:24.866780 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:25.172599 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:25.172403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" event={"ID":"2c1cf753-cfd4-4130-b5df-82d3c12f2fbd","Type":"ContainerStarted","Data":"7845925353c3e323d38d0fd35c497496c5b28f1395b119d14bc4d498c97083ce"} Sep 11 11:34:25.172712 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:25.172659 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:25.174555 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:25.174515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerStarted","Data":"14691fe2c6a7a00d48f42a7ef534e6720e41ccf7f1ecf01c11c1e8e8750a7ca1"} Sep 11 11:34:25.188409 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:25.188390 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:25.197340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:25.197300 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" podStartSLOduration=8.478337811 podStartE2EDuration="26.197286414s" totalImagesPullingTime="17.718948603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:25.196414236 +0000 UTC m=+26.831007381" watchObservedRunningTime="2026-09-11 11:34:25.197286414 +0000 UTC m=+26.831879547" Sep 11 11:34:25.866839 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:25.866806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:25.867414 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:25.866811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:25.867414 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:25.866908 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:25.867414 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:25.866988 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:26.179079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.178964 2580 generic.go:356] "Generic (PLEG): container finished" podID="1932cdb7-696f-48f4-8519-6e624ae80d71" containerID="14691fe2c6a7a00d48f42a7ef534e6720e41ccf7f1ecf01c11c1e8e8750a7ca1" exitCode=0 Sep 11 11:34:26.179221 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.179079 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerDied","Data":"14691fe2c6a7a00d48f42a7ef534e6720e41ccf7f1ecf01c11c1e8e8750a7ca1"} Sep 11 11:34:26.179221 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.179127 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 11:34:26.179647 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.179612 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:26.198761 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.198737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:26.866930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.866436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:26.866930 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:26.866564 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:26.871329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.870968 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8pb6s"] Sep 11 11:34:26.871329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.871017 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7nn5q"] Sep 11 11:34:26.871329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.871060 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s57lp"] Sep 11 11:34:26.871329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.871113 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:26.871329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:26.871111 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:26.871329 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:26.871205 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:26.871329 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:26.871288 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:27.184235 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:27.184201 2580 generic.go:356] "Generic (PLEG): container finished" podID="1932cdb7-696f-48f4-8519-6e624ae80d71" containerID="cba31b7bb1900540f938158d17f186f5dee553679346895dc9add158259b682d" exitCode=0 Sep 11 11:34:27.184419 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:27.184279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerDied","Data":"cba31b7bb1900540f938158d17f186f5dee553679346895dc9add158259b682d"} Sep 11 11:34:27.184419 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:27.184295 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:27.184520 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:27.184444 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 11:34:27.184570 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:27.184509 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:27.804169 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:27.804135 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:28.189062 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:28.188969 2580 generic.go:356] "Generic (PLEG): container finished" podID="1932cdb7-696f-48f4-8519-6e624ae80d71" containerID="8e983b0a76c2e1f5bbee347780bc3e463763f5e9ca27ae1ebbd773d02f5a96a3" exitCode=0 Sep 11 11:34:28.189432 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:28.189059 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerDied","Data":"8e983b0a76c2e1f5bbee347780bc3e463763f5e9ca27ae1ebbd773d02f5a96a3"} Sep 11 11:34:28.867395 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:28.867361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:28.867579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:28.867459 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:28.867579 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:28.867471 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:28.867579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:28.867523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:28.867748 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:28.867628 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:28.867748 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:28.867705 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:29.204505 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:29.204249 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" podUID="2c1cf753-cfd4-4130-b5df-82d3c12f2fbd" containerName="ovnkube-controller" probeResult="failure" output="" Sep 11 11:34:30.867097 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:30.866825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:30.867097 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:30.866850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:30.867097 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:30.866850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:30.867097 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:30.866951 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8pb6s" podUID="3d689159-7ed7-4c57-af29-1b64ac0491b6" Sep 11 11:34:30.867097 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:30.867007 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nn5q" podUID="ebef619d-9d2e-4a27-9b1c-300daad95756" Sep 11 11:34:30.867654 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:30.867139 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s57lp" podUID="6df5f2cf-051f-4ab9-a41d-7f0305cc486f" Sep 11 11:34:32.524465 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.524428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:32.524465 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.524478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:32.524989 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.524599 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:32.524989 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.524602 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:32.524989 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.524664 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs podName:6df5f2cf-051f-4ab9-a41d-7f0305cc486f nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.52464679 +0000 UTC m=+66.159239921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs") pod "network-metrics-daemon-s57lp" (UID: "6df5f2cf-051f-4ab9-a41d-7f0305cc486f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 11:34:32.524989 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.524678 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret podName:ebef619d-9d2e-4a27-9b1c-300daad95756 nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.524672309 +0000 UTC m=+66.159265421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret") pod "global-pull-secret-syncer-7nn5q" (UID: "ebef619d-9d2e-4a27-9b1c-300daad95756") : object "kube-system"/"original-pull-secret" not registered Sep 11 11:34:32.625276 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.625198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:32.625429 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.625388 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 11:34:32.625429 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.625418 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 11:34:32.625429 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.625429 2580 projected.go:196] Error preparing data for projected volume kube-api-access-ln89l for pod openshift-network-diagnostics/network-check-target-8pb6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:32.625565 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.625494 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l podName:3d689159-7ed7-4c57-af29-1b64ac0491b6 nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.625475652 +0000 UTC m=+66.260068785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ln89l" (UniqueName: "kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l") pod "network-check-target-8pb6s" (UID: "3d689159-7ed7-4c57-af29-1b64ac0491b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 11:34:32.628155 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.628131 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-183.ec2.internal" event="NodeReady" Sep 11 11:34:32.628286 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.628260 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 11:34:32.666539 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.666500 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w"] Sep 11 11:34:32.714457 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.714427 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6fcb677dbb-2nz2z"] Sep 11 11:34:32.714622 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.714547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.718149 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.718124 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.718334 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.718310 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.718500 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.718404 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 11:34:32.718655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.718640 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2clhp\"" Sep 11 11:34:32.724453 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.724426 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 11:34:32.732050 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.731338 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-ldgwv"] Sep 11 11:34:32.732050 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.731463 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.733676 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.733568 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 11:34:32.733957 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.733935 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 11:34:32.734519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.734502 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.735061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.734784 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 11:34:32.735061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.734805 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.735061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.734920 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 11:34:32.735061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.734975 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p6ddf\"" Sep 11 11:34:32.753992 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.753969 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-x7dfh"] Sep 11 11:34:32.754116 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.754102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.756406 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.756384 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 11:34:32.756533 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.756517 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.756612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.756565 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8d5k5\"" Sep 11 11:34:32.756612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.756588 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.756725 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.756653 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 11:34:32.763326 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.763308 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 11:34:32.776577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.776561 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr"] Sep 11 11:34:32.776697 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.776684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.780373 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.780347 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 11:34:32.780711 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.780694 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vl6sx\"" Sep 11 11:34:32.781462 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.780963 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 11:34:32.781462 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.781249 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.781883 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.781864 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.787667 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.787649 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 11:34:32.791176 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.791162 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f854f6d-vb2gm"] Sep 11 11:34:32.791285 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.791271 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:32.797051 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.797011 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 11:34:32.797136 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.797069 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 11:34:32.797230 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.797198 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.797315 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.797219 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.797369 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.797314 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mm5m7\"" Sep 11 11:34:32.810100 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.810081 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc"] Sep 11 11:34:32.810224 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.810210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.813028 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.813012 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 11:34:32.813028 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.813019 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mms9n\"" Sep 11 11:34:32.813028 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.813013 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 11:34:32.813195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.813019 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 11:34:32.819161 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.819146 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 11:34:32.826867 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.826849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df17a3aa-2361-405d-8165-4f5dd30e4318-trusted-ca\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.826946 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.826895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.826946 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.826913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a4679a-bf25-4d80-810e-4df2cf36dd98-service-ca-bundle\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.826946 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.826928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/06d8735e-012c-40ae-99db-84fba64b0bc2-kube-api-access-2xkn5\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.827071 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.826960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv26z\" (UniqueName: \"kubernetes.io/projected/df17a3aa-2361-405d-8165-4f5dd30e4318-kube-api-access-nv26z\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.827071 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3a4679a-bf25-4d80-810e-4df2cf36dd98-snapshots\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.827145 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jwj\" (UniqueName: \"kubernetes.io/projected/064268d3-1d9f-48d1-8c10-da03ed41b29c-kube-api-access-z5jwj\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.827145 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a4679a-bf25-4d80-810e-4df2cf36dd98-serving-cert\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.827145 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-default-certificate\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.827262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.827262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df17a3aa-2361-405d-8165-4f5dd30e4318-config\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.827262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df17a3aa-2361-405d-8165-4f5dd30e4318-serving-cert\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.827262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a4679a-bf25-4d80-810e-4df2cf36dd98-tmp\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.827262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a4679a-bf25-4d80-810e-4df2cf36dd98-trusted-ca-bundle\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.827448 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-stats-auth\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.827448 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06d8735e-012c-40ae-99db-84fba64b0bc2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.827448 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.827448 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.827345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdwx\" (UniqueName: \"kubernetes.io/projected/e3a4679a-bf25-4d80-810e-4df2cf36dd98-kube-api-access-hsdwx\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.830769 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.830754 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4"] Sep 11 11:34:32.830885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.830870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" Sep 11 11:34:32.833384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.833364 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.833384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.833378 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-brmm4\"" Sep 11 11:34:32.833384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.833368 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.840848 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.840830 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv"] Sep 11 11:34:32.840951 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.840938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:32.844392 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.844373 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.844731 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.844701 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 11:34:32.844810 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.844746 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.844911 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.844716 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dz6lt\"" Sep 11 11:34:32.845111 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.845098 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 11:34:32.859211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.859196 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w"] Sep 11 11:34:32.859300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.859244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr"] Sep 11 11:34:32.859300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.859255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:32.859396 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.859260 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb"] Sep 11 11:34:32.861459 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.861441 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v7n5d\"" Sep 11 11:34:32.861535 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.861477 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.861535 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.861481 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 11:34:32.861632 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.861513 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.879437 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879360 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fcb677dbb-2nz2z"] Sep 11 11:34:32.879437 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:34:32.879577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:34:32.879577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879488 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc"] Sep 11 11:34:32.879577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879512 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4"] Sep 11 11:34:32.879577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:34:32.879577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879534 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f854f6d-vb2gm"] Sep 11 11:34:32.879577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.879544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" Sep 11 11:34:32.882058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.882015 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tk98p\"" Sep 11 11:34:32.882454 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.882434 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 11:34:32.882706 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.882682 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 11:34:32.882776 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.882719 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.883009 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.882991 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b45p4\"" Sep 11 11:34:32.883586 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.883187 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.883752 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.883680 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lcdlp\"" Sep 11 11:34:32.885422 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.885400 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-x7dfh"] Sep 11 11:34:32.885509 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.885446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-ldgwv"] Sep 11 11:34:32.885509 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.885476 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2gldc"] Sep 11 11:34:32.901409 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.901392 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jd48q"] Sep 11 11:34:32.901542 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.901524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:32.904260 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.904242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 11:34:32.904558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.904542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-glmqf\"" Sep 11 11:34:32.904624 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.904568 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 11:34:32.904624 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.904542 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 11:34:32.915486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.915467 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv"] Sep 11 11:34:32.915569 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.915510 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb"] Sep 11 11:34:32.915569 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.915532 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gldc"] Sep 11 11:34:32.915569 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.915551 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jd48q"] Sep 11 11:34:32.915569 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.915564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:32.917704 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.917690 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 11:34:32.917966 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.917950 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 11:34:32.917966 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.917959 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nnmkw\"" Sep 11 11:34:32.928386 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-default-certificate\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.928537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jwj\" (UniqueName: \"kubernetes.io/projected/064268d3-1d9f-48d1-8c10-da03ed41b29c-kube-api-access-z5jwj\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.928537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a4679a-bf25-4d80-810e-4df2cf36dd98-serving-cert\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.928537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2nq5\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-kube-api-access-j2nq5\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.928537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhk22\" (UniqueName: \"kubernetes.io/projected/ec29893b-f200-4103-8d1d-63bbe5194a1a-kube-api-access-jhk22\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:32.928537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-bound-sa-token\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.928537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.928537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df17a3aa-2361-405d-8165-4f5dd30e4318-serving-cert\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a4679a-bf25-4d80-810e-4df2cf36dd98-trusted-ca-bundle\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdwx\" (UniqueName: \"kubernetes.io/projected/e3a4679a-bf25-4d80-810e-4df2cf36dd98-kube-api-access-hsdwx\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-certificates\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.928642 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5ecb8de-47a3-4c7a-90c8-099eb148851c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.928731 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls podName:06d8735e-012c-40ae-99db-84fba64b0bc2 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:33.428711719 +0000 UTC m=+35.063304850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zld2w" (UID: "06d8735e-012c-40ae-99db-84fba64b0bc2") : secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5ecb8de-47a3-4c7a-90c8-099eb148851c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a4679a-bf25-4d80-810e-4df2cf36dd98-service-ca-bundle\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv26z\" (UniqueName: \"kubernetes.io/projected/df17a3aa-2361-405d-8165-4f5dd30e4318-kube-api-access-nv26z\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec29893b-f200-4103-8d1d-63bbe5194a1a-config\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:32.928890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/06d8735e-012c-40ae-99db-84fba64b0bc2-kube-api-access-2xkn5\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.928955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-image-registry-private-configuration\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2f7r\" (UniqueName: \"kubernetes.io/projected/e5ecb8de-47a3-4c7a-90c8-099eb148851c-kube-api-access-s2f7r\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06d8735e-012c-40ae-99db-84fba64b0bc2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.929081 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3a4679a-bf25-4d80-810e-4df2cf36dd98-snapshots\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.929137 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:33.429106864 +0000 UTC m=+35.063699974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : secret "router-metrics-certs-default" not found Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df17a3aa-2361-405d-8165-4f5dd30e4318-config\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec29893b-f200-4103-8d1d-63bbe5194a1a-serving-cert\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a4679a-bf25-4d80-810e-4df2cf36dd98-tmp\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-trusted-ca\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-stats-auth\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a4679a-bf25-4d80-810e-4df2cf36dd98-service-ca-bundle\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5h7\" (UniqueName: \"kubernetes.io/projected/4745913e-1dcb-4b94-a93d-99069e5b4182-kube-api-access-rn5h7\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:32.929480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df17a3aa-2361-405d-8165-4f5dd30e4318-trusted-ca\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3a4679a-bf25-4d80-810e-4df2cf36dd98-trusted-ca-bundle\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-installation-pull-secrets\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8nc\" (UniqueName: \"kubernetes.io/projected/a2713d0e-8325-4a04-920e-7ed39b1afb9e-kube-api-access-mg8nc\") pod \"volume-data-source-validator-86695b7cc7-4f8rc\" (UID: \"a2713d0e-8325-4a04-920e-7ed39b1afb9e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80bf58f5-0918-4934-a5f2-f734cb77b81f-ca-trust-extracted\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e3a4679a-bf25-4d80-810e-4df2cf36dd98-snapshots\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:32.929808 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:33.429787466 +0000 UTC m=+35.064380600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : configmap references non-existent config key: service-ca.crt Sep 11 11:34:32.930171 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.929961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a4679a-bf25-4d80-810e-4df2cf36dd98-tmp\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.930544 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.930383 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/06d8735e-012c-40ae-99db-84fba64b0bc2-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.930544 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.930441 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df17a3aa-2361-405d-8165-4f5dd30e4318-config\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.930675 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.930655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df17a3aa-2361-405d-8165-4f5dd30e4318-trusted-ca\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.933190 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.933168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df17a3aa-2361-405d-8165-4f5dd30e4318-serving-cert\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.933269 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.933195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-default-certificate\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.933269 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.933196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-stats-auth\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.933333 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.933323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a4679a-bf25-4d80-810e-4df2cf36dd98-serving-cert\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.939765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.939740 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdwx\" (UniqueName: \"kubernetes.io/projected/e3a4679a-bf25-4d80-810e-4df2cf36dd98-kube-api-access-hsdwx\") pod \"insights-operator-5844b856db-x7dfh\" (UID: \"e3a4679a-bf25-4d80-810e-4df2cf36dd98\") " pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:32.939933 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.939916 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jwj\" (UniqueName: \"kubernetes.io/projected/064268d3-1d9f-48d1-8c10-da03ed41b29c-kube-api-access-z5jwj\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:32.940697 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.940672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xkn5\" (UniqueName: \"kubernetes.io/projected/06d8735e-012c-40ae-99db-84fba64b0bc2-kube-api-access-2xkn5\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:32.941816 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.941796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv26z\" (UniqueName: \"kubernetes.io/projected/df17a3aa-2361-405d-8165-4f5dd30e4318-kube-api-access-nv26z\") pod \"console-operator-696884fb9c-ldgwv\" (UID: \"df17a3aa-2361-405d-8165-4f5dd30e4318\") " pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:32.946885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:32.946855 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:33.030337 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-certificates\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.030337 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5ecb8de-47a3-4c7a-90c8-099eb148851c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:33.030512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:33.030512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5ecb8de-47a3-4c7a-90c8-099eb148851c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:33.030512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec29893b-f200-4103-8d1d-63bbe5194a1a-config\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:33.030512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7jn\" (UniqueName: \"kubernetes.io/projected/033fa8a3-2baf-401c-9792-759c21d55ea0-kube-api-access-lm7jn\") pod \"ingress-canary-2gldc\" (UID: \"033fa8a3-2baf-401c-9792-759c21d55ea0\") " pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:33.030512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-image-registry-private-configuration\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.030512 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.030482 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 11:34:33.030512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2f7r\" (UniqueName: \"kubernetes.io/projected/e5ecb8de-47a3-4c7a-90c8-099eb148851c-kube-api-access-s2f7r\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40526ae2-253a-434c-bff9-5508117eb6c0-config-volume\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.030569 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls podName:4745913e-1dcb-4b94-a93d-99069e5b4182 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:33.530551665 +0000 UTC m=+35.165144798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wszbv" (UID: "4745913e-1dcb-4b94-a93d-99069e5b4182") : secret "samples-operator-tls" not found Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec29893b-f200-4103-8d1d-63bbe5194a1a-serving-cert\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-trusted-ca\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5h7\" (UniqueName: \"kubernetes.io/projected/4745913e-1dcb-4b94-a93d-99069e5b4182-kube-api-access-rn5h7\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzfs\" (UniqueName: \"kubernetes.io/projected/40526ae2-253a-434c-bff9-5508117eb6c0-kube-api-access-dlzfs\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030769 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-installation-pull-secrets\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030799 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8nc\" (UniqueName: \"kubernetes.io/projected/a2713d0e-8325-4a04-920e-7ed39b1afb9e-kube-api-access-mg8nc\") pod \"volume-data-source-validator-86695b7cc7-4f8rc\" (UID: \"a2713d0e-8325-4a04-920e-7ed39b1afb9e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" Sep 11 11:34:33.030838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40526ae2-253a-434c-bff9-5508117eb6c0-tmp-dir\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80bf58f5-0918-4934-a5f2-f734cb77b81f-ca-trust-extracted\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/033fa8a3-2baf-401c-9792-759c21d55ea0-cert\") pod \"ingress-canary-2gldc\" (UID: \"033fa8a3-2baf-401c-9792-759c21d55ea0\") " pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzph\" (UniqueName: \"kubernetes.io/projected/11b5bbfb-ef97-484c-84fc-a2d237fb2d54-kube-api-access-gfzph\") pod \"network-check-source-6f8b768455-mg8gb\" (UID: \"11b5bbfb-ef97-484c-84fc-a2d237fb2d54\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e5ecb8de-47a3-4c7a-90c8-099eb148851c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2nq5\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-kube-api-access-j2nq5\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.030974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-certificates\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.031006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhk22\" (UniqueName: \"kubernetes.io/projected/ec29893b-f200-4103-8d1d-63bbe5194a1a-kube-api-access-jhk22\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.031022 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.031062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-bound-sa-token\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.031067 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f854f6d-vb2gm: secret "image-registry-tls" not found Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.031157 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls podName:80bf58f5-0918-4934-a5f2-f734cb77b81f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:33.531140608 +0000 UTC m=+35.165733719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls") pod "image-registry-84f854f6d-vb2gm" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f") : secret "image-registry-tls" not found Sep 11 11:34:33.031384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.031162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ec29893b-f200-4103-8d1d-63bbe5194a1a-config\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:33.032029 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.032008 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-trusted-ca\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.033162 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.033141 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ec29893b-f200-4103-8d1d-63bbe5194a1a-serving-cert\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:33.033262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.033245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-image-registry-private-configuration\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.033658 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.033635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e5ecb8de-47a3-4c7a-90c8-099eb148851c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:33.033892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.033877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-installation-pull-secrets\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.039442 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.039423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80bf58f5-0918-4934-a5f2-f734cb77b81f-ca-trust-extracted\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.042838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.042821 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8nc\" (UniqueName: \"kubernetes.io/projected/a2713d0e-8325-4a04-920e-7ed39b1afb9e-kube-api-access-mg8nc\") pod \"volume-data-source-validator-86695b7cc7-4f8rc\" (UID: \"a2713d0e-8325-4a04-920e-7ed39b1afb9e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" Sep 11 11:34:33.044285 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.044258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5h7\" (UniqueName: \"kubernetes.io/projected/4745913e-1dcb-4b94-a93d-99069e5b4182-kube-api-access-rn5h7\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:33.044453 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.044434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2nq5\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-kube-api-access-j2nq5\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.045020 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.045002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-bound-sa-token\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.045423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.045403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2f7r\" (UniqueName: \"kubernetes.io/projected/e5ecb8de-47a3-4c7a-90c8-099eb148851c-kube-api-access-s2f7r\") pod \"kube-storage-version-migrator-operator-6c8f84f897-rxcpr\" (UID: \"e5ecb8de-47a3-4c7a-90c8-099eb148851c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:33.045796 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.045782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhk22\" (UniqueName: \"kubernetes.io/projected/ec29893b-f200-4103-8d1d-63bbe5194a1a-kube-api-access-jhk22\") pod \"service-ca-operator-cf7c776b7-d9hh4\" (UID: \"ec29893b-f200-4103-8d1d-63bbe5194a1a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:33.047332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.047312 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:33.062139 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.062120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:33.087405 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.087386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-x7dfh" Sep 11 11:34:33.098867 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.098851 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" Sep 11 11:34:33.132461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.132400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40526ae2-253a-434c-bff9-5508117eb6c0-tmp-dir\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.132461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.132434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/033fa8a3-2baf-401c-9792-759c21d55ea0-cert\") pod \"ingress-canary-2gldc\" (UID: \"033fa8a3-2baf-401c-9792-759c21d55ea0\") " pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:33.132461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.132457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzph\" (UniqueName: \"kubernetes.io/projected/11b5bbfb-ef97-484c-84fc-a2d237fb2d54-kube-api-access-gfzph\") pod \"network-check-source-6f8b768455-mg8gb\" (UID: \"11b5bbfb-ef97-484c-84fc-a2d237fb2d54\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" Sep 11 11:34:33.132886 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.132862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7jn\" (UniqueName: \"kubernetes.io/projected/033fa8a3-2baf-401c-9792-759c21d55ea0-kube-api-access-lm7jn\") pod \"ingress-canary-2gldc\" (UID: \"033fa8a3-2baf-401c-9792-759c21d55ea0\") " pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:33.133076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.132914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40526ae2-253a-434c-bff9-5508117eb6c0-config-volume\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.133076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.132945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.133076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.132989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzfs\" (UniqueName: \"kubernetes.io/projected/40526ae2-253a-434c-bff9-5508117eb6c0-kube-api-access-dlzfs\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.133334 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.133123 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:34:33.133334 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.133140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40526ae2-253a-434c-bff9-5508117eb6c0-tmp-dir\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.133334 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.133177 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls podName:40526ae2-253a-434c-bff9-5508117eb6c0 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:33.633158101 +0000 UTC m=+35.267751214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls") pod "dns-default-jd48q" (UID: "40526ae2-253a-434c-bff9-5508117eb6c0") : secret "dns-default-metrics-tls" not found Sep 11 11:34:33.133571 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.133448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40526ae2-253a-434c-bff9-5508117eb6c0-config-volume\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.135338 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.135320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/033fa8a3-2baf-401c-9792-759c21d55ea0-cert\") pod \"ingress-canary-2gldc\" (UID: \"033fa8a3-2baf-401c-9792-759c21d55ea0\") " pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:33.138113 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.138099 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" Sep 11 11:34:33.142377 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.142354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzfs\" (UniqueName: \"kubernetes.io/projected/40526ae2-253a-434c-bff9-5508117eb6c0-kube-api-access-dlzfs\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.142607 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.142584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzph\" (UniqueName: \"kubernetes.io/projected/11b5bbfb-ef97-484c-84fc-a2d237fb2d54-kube-api-access-gfzph\") pod \"network-check-source-6f8b768455-mg8gb\" (UID: \"11b5bbfb-ef97-484c-84fc-a2d237fb2d54\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" Sep 11 11:34:33.142705 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.142626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7jn\" (UniqueName: \"kubernetes.io/projected/033fa8a3-2baf-401c-9792-759c21d55ea0-kube-api-access-lm7jn\") pod \"ingress-canary-2gldc\" (UID: \"033fa8a3-2baf-401c-9792-759c21d55ea0\") " pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:33.148542 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.148348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" Sep 11 11:34:33.209186 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.208864 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" Sep 11 11:34:33.214733 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.213830 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gldc" Sep 11 11:34:33.259014 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.258564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:33.331327 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.331296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:33.435773 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.435741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:33.435897 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.435846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:33.435897 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.435872 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:34.435847547 +0000 UTC m=+36.070440657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : configmap references non-existent config key: service-ca.crt Sep 11 11:34:33.436011 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.435929 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:33.436011 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.435960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:33.436011 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.435975 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls podName:06d8735e-012c-40ae-99db-84fba64b0bc2 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:34.435960939 +0000 UTC m=+36.070554067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zld2w" (UID: "06d8735e-012c-40ae-99db-84fba64b0bc2") : secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:33.436144 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.436060 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 11:34:33.436144 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.436103 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:34.436090779 +0000 UTC m=+36.070683904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : secret "router-metrics-certs-default" not found Sep 11 11:34:33.438617 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.438593 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc"] Sep 11 11:34:33.445404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.445383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-x7dfh"] Sep 11 11:34:33.446316 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.446298 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-ldgwv"] Sep 11 11:34:33.459691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.459667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr"] Sep 11 11:34:33.468486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.468466 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4"] Sep 11 11:34:33.528556 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.528527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gldc"] Sep 11 11:34:33.535272 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.535241 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb"] Sep 11 11:34:33.537565 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.537540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:33.537672 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.537621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:33.537767 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.537723 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 11:34:33.537767 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.537726 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:34:33.537767 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.537746 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f854f6d-vb2gm: secret "image-registry-tls" not found Sep 11 11:34:33.537850 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.537776 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls podName:4745913e-1dcb-4b94-a93d-99069e5b4182 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:34.537760794 +0000 UTC m=+36.172353925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wszbv" (UID: "4745913e-1dcb-4b94-a93d-99069e5b4182") : secret "samples-operator-tls" not found Sep 11 11:34:33.537850 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.537792 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls podName:80bf58f5-0918-4934-a5f2-f734cb77b81f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:34.537784972 +0000 UTC m=+36.172378086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls") pod "image-registry-84f854f6d-vb2gm" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f") : secret "image-registry-tls" not found Sep 11 11:34:33.540473 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:33.540450 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2713d0e_8325_4a04_920e_7ed39b1afb9e.slice/crio-64e4e008cdb0b178662402048bfd9bc801ae4c25f63478ef12ad5f93d06552be WatchSource:0}: Error finding container 64e4e008cdb0b178662402048bfd9bc801ae4c25f63478ef12ad5f93d06552be: Status 404 returned error can't find the container with id 64e4e008cdb0b178662402048bfd9bc801ae4c25f63478ef12ad5f93d06552be Sep 11 11:34:33.541131 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:33.541098 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b5bbfb_ef97_484c_84fc_a2d237fb2d54.slice/crio-24d52d8167aa1811a5cb7aa9edf7e7d61539c84409242f453d140eef9b504dd8 WatchSource:0}: Error finding container 24d52d8167aa1811a5cb7aa9edf7e7d61539c84409242f453d140eef9b504dd8: Status 404 returned error can't find the container with id 24d52d8167aa1811a5cb7aa9edf7e7d61539c84409242f453d140eef9b504dd8 Sep 11 11:34:33.542683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:33.542666 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a4679a_bf25_4d80_810e_4df2cf36dd98.slice/crio-05e303b896ae35ac787414f9bef75b94cf7967c4c0c434234aeaa90a4f54f459 WatchSource:0}: Error finding container 05e303b896ae35ac787414f9bef75b94cf7967c4c0c434234aeaa90a4f54f459: Status 404 returned error can't find the container with id 05e303b896ae35ac787414f9bef75b94cf7967c4c0c434234aeaa90a4f54f459 Sep 11 11:34:33.543011 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:33.542962 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf17a3aa_2361_405d_8165_4f5dd30e4318.slice/crio-299919ccd11529d8c445211ff11695f8b8ed477ba9bb9ad5eb288925038e88a7 WatchSource:0}: Error finding container 299919ccd11529d8c445211ff11695f8b8ed477ba9bb9ad5eb288925038e88a7: Status 404 returned error can't find the container with id 299919ccd11529d8c445211ff11695f8b8ed477ba9bb9ad5eb288925038e88a7 Sep 11 11:34:33.544455 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:33.544411 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec29893b_f200_4103_8d1d_63bbe5194a1a.slice/crio-84ca0d08de1fdde628f088d11bcffe9921bebad30aff36029b1668f2dc94da8a WatchSource:0}: Error finding container 84ca0d08de1fdde628f088d11bcffe9921bebad30aff36029b1668f2dc94da8a: Status 404 returned error can't find the container with id 84ca0d08de1fdde628f088d11bcffe9921bebad30aff36029b1668f2dc94da8a Sep 11 11:34:33.545367 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:33.545344 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033fa8a3_2baf_401c_9792_759c21d55ea0.slice/crio-529ba959ed8f43531c178078993969b122952f0f7fbf72985e50996c3423629a WatchSource:0}: Error finding container 529ba959ed8f43531c178078993969b122952f0f7fbf72985e50996c3423629a: Status 404 returned error can't find the container with id 529ba959ed8f43531c178078993969b122952f0f7fbf72985e50996c3423629a Sep 11 11:34:33.548194 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:33.547289 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ecb8de_47a3_4c7a_90c8_099eb148851c.slice/crio-f4dea8a76d10a1d0fdf26cb8300358dd4c82b9331516fda1bbd51c66e40651d8 WatchSource:0}: Error finding container f4dea8a76d10a1d0fdf26cb8300358dd4c82b9331516fda1bbd51c66e40651d8: Status 404 returned error can't find the container with id f4dea8a76d10a1d0fdf26cb8300358dd4c82b9331516fda1bbd51c66e40651d8 Sep 11 11:34:33.638449 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.638416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:33.638563 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.638541 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:34:33.638618 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:33.638594 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls podName:40526ae2-253a-434c-bff9-5508117eb6c0 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:34.638578535 +0000 UTC m=+36.273171648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls") pod "dns-default-jd48q" (UID: "40526ae2-253a-434c-bff9-5508117eb6c0") : secret "dns-default-metrics-tls" not found Sep 11 11:34:33.971533 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:33.971501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:34.077680 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.077642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:34.209865 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.209822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" event={"ID":"a2713d0e-8325-4a04-920e-7ed39b1afb9e","Type":"ContainerStarted","Data":"64e4e008cdb0b178662402048bfd9bc801ae4c25f63478ef12ad5f93d06552be"} Sep 11 11:34:34.212255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.212189 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" event={"ID":"ec29893b-f200-4103-8d1d-63bbe5194a1a","Type":"ContainerStarted","Data":"84ca0d08de1fdde628f088d11bcffe9921bebad30aff36029b1668f2dc94da8a"} Sep 11 11:34:34.215654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.215579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" event={"ID":"e5ecb8de-47a3-4c7a-90c8-099eb148851c","Type":"ContainerStarted","Data":"f4dea8a76d10a1d0fdf26cb8300358dd4c82b9331516fda1bbd51c66e40651d8"} Sep 11 11:34:34.217143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.217117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x7dfh" event={"ID":"e3a4679a-bf25-4d80-810e-4df2cf36dd98","Type":"ContainerStarted","Data":"05e303b896ae35ac787414f9bef75b94cf7967c4c0c434234aeaa90a4f54f459"} Sep 11 11:34:34.218472 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.218443 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" event={"ID":"df17a3aa-2361-405d-8165-4f5dd30e4318","Type":"ContainerStarted","Data":"299919ccd11529d8c445211ff11695f8b8ed477ba9bb9ad5eb288925038e88a7"} Sep 11 11:34:34.220310 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.220259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gldc" event={"ID":"033fa8a3-2baf-401c-9792-759c21d55ea0","Type":"ContainerStarted","Data":"529ba959ed8f43531c178078993969b122952f0f7fbf72985e50996c3423629a"} Sep 11 11:34:34.221739 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.221655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" event={"ID":"11b5bbfb-ef97-484c-84fc-a2d237fb2d54","Type":"ContainerStarted","Data":"24d52d8167aa1811a5cb7aa9edf7e7d61539c84409242f453d140eef9b504dd8"} Sep 11 11:34:34.225940 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.225901 2580 generic.go:356] "Generic (PLEG): container finished" podID="1932cdb7-696f-48f4-8519-6e624ae80d71" containerID="d284fc626fdfb264d4caba8dfc1a9c0623ec88c44382ad88d01d8ff8ba1307ec" exitCode=0 Sep 11 11:34:34.226064 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.225950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerDied","Data":"d284fc626fdfb264d4caba8dfc1a9c0623ec88c44382ad88d01d8ff8ba1307ec"} Sep 11 11:34:34.446221 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.445942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:34.446368 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.446296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:34.446439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.446398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:34.446510 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.446159 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 11:34:34.446587 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.446573 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:34.446640 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.446588 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:36.446569258 +0000 UTC m=+38.081162369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : secret "router-metrics-certs-default" not found Sep 11 11:34:34.446640 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.446632 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls podName:06d8735e-012c-40ae-99db-84fba64b0bc2 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:36.446616188 +0000 UTC m=+38.081209304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zld2w" (UID: "06d8735e-012c-40ae-99db-84fba64b0bc2") : secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:34.446751 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.446703 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:36.44669199 +0000 UTC m=+38.081285100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : configmap references non-existent config key: service-ca.crt Sep 11 11:34:34.547172 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.547062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:34.547172 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.547153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:34.547700 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.547299 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 11:34:34.547700 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.547361 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls podName:4745913e-1dcb-4b94-a93d-99069e5b4182 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:36.547344165 +0000 UTC m=+38.181937293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wszbv" (UID: "4745913e-1dcb-4b94-a93d-99069e5b4182") : secret "samples-operator-tls" not found Sep 11 11:34:34.547815 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.547764 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:34:34.547815 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.547779 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f854f6d-vb2gm: secret "image-registry-tls" not found Sep 11 11:34:34.547917 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.547826 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls podName:80bf58f5-0918-4934-a5f2-f734cb77b81f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:36.547810683 +0000 UTC m=+38.182403808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls") pod "image-registry-84f854f6d-vb2gm" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f") : secret "image-registry-tls" not found Sep 11 11:34:34.648277 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:34.648241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:34.648603 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.648531 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:34:34.648603 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:34.648595 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls podName:40526ae2-253a-434c-bff9-5508117eb6c0 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:36.648577748 +0000 UTC m=+38.283170861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls") pod "dns-default-jd48q" (UID: "40526ae2-253a-434c-bff9-5508117eb6c0") : secret "dns-default-metrics-tls" not found Sep 11 11:34:35.238508 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:35.238446 2580 generic.go:356] "Generic (PLEG): container finished" podID="1932cdb7-696f-48f4-8519-6e624ae80d71" containerID="bbb19020f1bcffd9ccee954983ed239185f2e079df108c4ffe42c4bdca79c762" exitCode=0 Sep 11 11:34:35.238739 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:35.238522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerDied","Data":"bbb19020f1bcffd9ccee954983ed239185f2e079df108c4ffe42c4bdca79c762"} Sep 11 11:34:35.693193 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:35.693114 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:36.287779 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.287726 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8"] Sep 11 11:34:36.309070 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.308439 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8"] Sep 11 11:34:36.309070 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.308573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:36.311745 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.311723 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 11:34:36.312108 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.312089 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gwc9z\"" Sep 11 11:34:36.312458 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.312441 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 11:34:36.431899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.431815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:36.466463 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.466428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:36.466618 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.466512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:36.466618 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.466552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:36.466739 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.466612 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 11:34:36.466739 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.466625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa5874ab-3b5c-4d44-8a50-ea744be7834d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:36.466739 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.466662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:36.466739 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.466705 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:40.466662792 +0000 UTC m=+42.101255919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : secret "router-metrics-certs-default" not found Sep 11 11:34:36.466947 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.466750 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:36.466947 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.466803 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls podName:06d8735e-012c-40ae-99db-84fba64b0bc2 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:40.466786696 +0000 UTC m=+42.101379826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zld2w" (UID: "06d8735e-012c-40ae-99db-84fba64b0bc2") : secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:36.466947 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.466898 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:40.466890086 +0000 UTC m=+42.101483197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : configmap references non-existent config key: service-ca.crt Sep 11 11:34:36.533698 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.533655 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:36.567720 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.567685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa5874ab-3b5c-4d44-8a50-ea744be7834d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:36.567885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.567778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:36.567948 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.567883 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:36.568001 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.567947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:36.568131 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.568100 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:34:36.568131 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.568121 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f854f6d-vb2gm: secret "image-registry-tls" not found Sep 11 11:34:36.568298 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.568167 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 11:34:36.568298 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.568174 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls podName:80bf58f5-0918-4934-a5f2-f734cb77b81f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:40.56815706 +0000 UTC m=+42.202750188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls") pod "image-registry-84f854f6d-vb2gm" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f") : secret "image-registry-tls" not found Sep 11 11:34:36.568298 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.568204 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls podName:4745913e-1dcb-4b94-a93d-99069e5b4182 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:40.568193712 +0000 UTC m=+42.202786823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wszbv" (UID: "4745913e-1dcb-4b94-a93d-99069e5b4182") : secret "samples-operator-tls" not found Sep 11 11:34:36.568298 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.568214 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 11:34:36.568298 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.568280 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert podName:fa5874ab-3b5c-4d44-8a50-ea744be7834d nodeName:}" failed. No retries permitted until 2026-09-11 11:34:37.068260357 +0000 UTC m=+38.702853483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7dsg8" (UID: "fa5874ab-3b5c-4d44-8a50-ea744be7834d") : secret "networking-console-plugin-cert" not found Sep 11 11:34:36.568558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.568471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fa5874ab-3b5c-4d44-8a50-ea744be7834d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:36.633292 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.633250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:36.668477 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:36.668451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:36.668665 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.668634 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:34:36.668727 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:36.668713 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls podName:40526ae2-253a-434c-bff9-5508117eb6c0 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:40.668691314 +0000 UTC m=+42.303284443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls") pod "dns-default-jd48q" (UID: "40526ae2-253a-434c-bff9-5508117eb6c0") : secret "dns-default-metrics-tls" not found Sep 11 11:34:37.072216 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:37.072179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:37.072578 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:37.072344 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 11:34:37.072578 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:37.072420 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert podName:fa5874ab-3b5c-4d44-8a50-ea744be7834d nodeName:}" failed. No retries permitted until 2026-09-11 11:34:38.07240477 +0000 UTC m=+39.706997881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7dsg8" (UID: "fa5874ab-3b5c-4d44-8a50-ea744be7834d") : secret "networking-console-plugin-cert" not found Sep 11 11:34:37.810139 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:37.809963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:37.911286 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:37.911214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" limit=1024 Sep 11 11:34:38.081019 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:38.080938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:38.081476 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:38.081263 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 11:34:38.081476 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:38.081330 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert podName:fa5874ab-3b5c-4d44-8a50-ea744be7834d nodeName:}" failed. No retries permitted until 2026-09-11 11:34:40.081311652 +0000 UTC m=+41.715904769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7dsg8" (UID: "fa5874ab-3b5c-4d44-8a50-ea744be7834d") : secret "networking-console-plugin-cert" not found Sep 11 11:34:40.098508 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:40.098469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:40.098900 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.098614 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 11:34:40.098900 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.098679 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert podName:fa5874ab-3b5c-4d44-8a50-ea744be7834d nodeName:}" failed. No retries permitted until 2026-09-11 11:34:44.098664344 +0000 UTC m=+45.733257455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7dsg8" (UID: "fa5874ab-3b5c-4d44-8a50-ea744be7834d") : secret "networking-console-plugin-cert" not found Sep 11 11:34:40.501812 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:40.501774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:40.502018 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:40.501879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:40.502018 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.501910 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:40.502018 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:40.501938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:40.502018 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.501992 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls podName:06d8735e-012c-40ae-99db-84fba64b0bc2 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:48.501969628 +0000 UTC m=+50.136562743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zld2w" (UID: "06d8735e-012c-40ae-99db-84fba64b0bc2") : secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:40.502223 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.502022 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 11:34:40.502223 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.502065 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:48.50204947 +0000 UTC m=+50.136642601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : configmap references non-existent config key: service-ca.crt Sep 11 11:34:40.502223 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.502084 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:34:48.502073958 +0000 UTC m=+50.136667074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : secret "router-metrics-certs-default" not found Sep 11 11:34:40.603488 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:40.603456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:40.603645 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:40.603533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:40.603645 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.603629 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:34:40.603729 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.603638 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 11:34:40.603729 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.603648 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f854f6d-vb2gm: secret "image-registry-tls" not found Sep 11 11:34:40.603729 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.603706 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls podName:80bf58f5-0918-4934-a5f2-f734cb77b81f nodeName:}" failed. No retries permitted until 2026-09-11 11:34:48.603692415 +0000 UTC m=+50.238285530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls") pod "image-registry-84f854f6d-vb2gm" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f") : secret "image-registry-tls" not found Sep 11 11:34:40.603729 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.603718 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls podName:4745913e-1dcb-4b94-a93d-99069e5b4182 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:48.603712793 +0000 UTC m=+50.238305904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wszbv" (UID: "4745913e-1dcb-4b94-a93d-99069e5b4182") : secret "samples-operator-tls" not found Sep 11 11:34:40.704361 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:40.704323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:40.704518 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.704467 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:34:40.704565 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:40.704534 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls podName:40526ae2-253a-434c-bff9-5508117eb6c0 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:48.70451377 +0000 UTC m=+50.339106898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls") pod "dns-default-jd48q" (UID: "40526ae2-253a-434c-bff9-5508117eb6c0") : secret "dns-default-metrics-tls" not found Sep 11 11:34:41.258088 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.258049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" event={"ID":"ec29893b-f200-4103-8d1d-63bbe5194a1a","Type":"ContainerStarted","Data":"351c681252c475792aae90e0e4ee6173b8f141260578aec3f191533cdc0743bb"} Sep 11 11:34:41.259933 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.259905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" event={"ID":"e5ecb8de-47a3-4c7a-90c8-099eb148851c","Type":"ContainerStarted","Data":"7f1079fbec01d60e036450a263f9aaeb1f3d9623ff16ee13371d4af5c2a52e68"} Sep 11 11:34:41.262080 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.261720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x7dfh" event={"ID":"e3a4679a-bf25-4d80-810e-4df2cf36dd98","Type":"ContainerStarted","Data":"f8d3787d6915a1f486303589d87b8fa6f4c495564c4438da8e8cea9fc7368488"} Sep 11 11:34:41.263789 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.263549 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" event={"ID":"df17a3aa-2361-405d-8165-4f5dd30e4318","Type":"ContainerStarted","Data":"34afd0b2117f29191d83217a1e01e84b35913ffb5cf15b0e6c5f9108df4f3608"} Sep 11 11:34:41.263789 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.263717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:41.265610 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.265335 2580 patch_prober.go:28] interesting pod/console-operator-696884fb9c-ldgwv container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" start-of-body= Sep 11 11:34:41.265610 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.265398 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" podUID="df17a3aa-2361-405d-8165-4f5dd30e4318" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.13:8443/readyz\": dial tcp 10.132.0.13:8443: connect: connection refused" Sep 11 11:34:41.265951 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.265884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" event={"ID":"11b5bbfb-ef97-484c-84fc-a2d237fb2d54","Type":"ContainerStarted","Data":"5d63f2a09270ca76c42d0a3dd5066df0bdee4befd84acedd861d6d51ccd1e69b"} Sep 11 11:34:41.277831 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.276500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" event={"ID":"1932cdb7-696f-48f4-8519-6e624ae80d71","Type":"ContainerStarted","Data":"177c34a90d093f5db96f47c358ce7a402a406bbafe0f7a90951095f979e20a79"} Sep 11 11:34:41.279922 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.279903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" event={"ID":"a2713d0e-8325-4a04-920e-7ed39b1afb9e","Type":"ContainerStarted","Data":"61047bdc3c478a92749c318a83257f26c752c36ee12e68ef031b7277a70278df"} Sep 11 11:34:41.299894 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.299845 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" podStartSLOduration=14.962564495 podStartE2EDuration="22.299823253s" totalImagesPullingTime="7.337258758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:41.279330773 +0000 UTC m=+42.913923910" watchObservedRunningTime="2026-09-11 11:34:41.299823253 +0000 UTC m=+42.934416383" Sep 11 11:34:41.323959 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.323897 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" podStartSLOduration=14.991370351 podStartE2EDuration="22.323878273s" totalImagesPullingTime="7.332507922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:41.300184749 +0000 UTC m=+42.934777883" watchObservedRunningTime="2026-09-11 11:34:41.323878273 +0000 UTC m=+42.958471410" Sep 11 11:34:41.353285 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.352941 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-x7dfh" podStartSLOduration=25.011558749 podStartE2EDuration="32.352921597s" totalImagesPullingTime="7.341362848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:41.324004487 +0000 UTC m=+42.958597621" watchObservedRunningTime="2026-09-11 11:34:41.352921597 +0000 UTC m=+42.987514731" Sep 11 11:34:41.353962 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.353589 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rqpd6" podStartSLOduration=13.536910819 podStartE2EDuration="42.353572881s" totalImagesPullingTime="28.816662062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:41.352339197 +0000 UTC m=+42.986932331" watchObservedRunningTime="2026-09-11 11:34:41.353572881 +0000 UTC m=+42.988166015" Sep 11 11:34:41.372585 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.372414 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mg8gb" podStartSLOduration=4.030503625 podStartE2EDuration="11.372394376s" totalImagesPullingTime="7.341890751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:41.371094941 +0000 UTC m=+43.005688076" watchObservedRunningTime="2026-09-11 11:34:41.372394376 +0000 UTC m=+43.006987510" Sep 11 11:34:41.392318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.392023 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" podStartSLOduration=15.051500321 podStartE2EDuration="22.392007171s" totalImagesPullingTime="7.34050685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:41.389559989 +0000 UTC m=+43.024153124" watchObservedRunningTime="2026-09-11 11:34:41.392007171 +0000 UTC m=+43.026600311" Sep 11 11:34:41.408471 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:41.408429 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-4f8rc" podStartSLOduration=15.888061016 podStartE2EDuration="22.408407819s" totalImagesPullingTime="6.520346803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:41.407448331 +0000 UTC m=+43.042041497" watchObservedRunningTime="2026-09-11 11:34:41.408407819 +0000 UTC m=+43.043000951" Sep 11 11:34:42.203256 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.203222 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62"] Sep 11 11:34:42.219103 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.219073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62"] Sep 11 11:34:42.219265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.219144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" Sep 11 11:34:42.221445 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.221415 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 11 11:34:42.221609 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.221576 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dgl74\"" Sep 11 11:34:42.221704 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.221608 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 11 11:34:42.286189 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.286160 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/0.log" Sep 11 11:34:42.286662 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.286201 2580 generic.go:356] "Generic (PLEG): container finished" podID="df17a3aa-2361-405d-8165-4f5dd30e4318" containerID="34afd0b2117f29191d83217a1e01e84b35913ffb5cf15b0e6c5f9108df4f3608" exitCode=255 Sep 11 11:34:42.286662 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.286282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" event={"ID":"df17a3aa-2361-405d-8165-4f5dd30e4318","Type":"ContainerDied","Data":"34afd0b2117f29191d83217a1e01e84b35913ffb5cf15b0e6c5f9108df4f3608"} Sep 11 11:34:42.286662 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.286532 2580 scope.go:117] "RemoveContainer" containerID="34afd0b2117f29191d83217a1e01e84b35913ffb5cf15b0e6c5f9108df4f3608" Sep 11 11:34:42.288196 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.288170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gldc" event={"ID":"033fa8a3-2baf-401c-9792-759c21d55ea0","Type":"ContainerStarted","Data":"3169bf033f8ce245ac11efedd4bb6c1cf73e2558930b9373da22c57f64dbb327"} Sep 11 11:34:42.319052 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.318998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jlp\" (UniqueName: \"kubernetes.io/projected/fea7ab92-bea1-4209-bd83-490414b5ebcd-kube-api-access-68jlp\") pod \"migrator-8b4d697c5-rpx62\" (UID: \"fea7ab92-bea1-4209-bd83-490414b5ebcd\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" Sep 11 11:34:42.347536 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.347482 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2gldc" podStartSLOduration=2.598900667 podStartE2EDuration="10.347459242s" totalImagesPullingTime="7.748558575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:42.345955583 +0000 UTC m=+43.980548716" watchObservedRunningTime="2026-09-11 11:34:42.347459242 +0000 UTC m=+43.982052376" Sep 11 11:34:42.421392 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.421066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jlp\" (UniqueName: \"kubernetes.io/projected/fea7ab92-bea1-4209-bd83-490414b5ebcd-kube-api-access-68jlp\") pod \"migrator-8b4d697c5-rpx62\" (UID: \"fea7ab92-bea1-4209-bd83-490414b5ebcd\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" Sep 11 11:34:42.432648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.432617 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jlp\" (UniqueName: \"kubernetes.io/projected/fea7ab92-bea1-4209-bd83-490414b5ebcd-kube-api-access-68jlp\") pod \"migrator-8b4d697c5-rpx62\" (UID: \"fea7ab92-bea1-4209-bd83-490414b5ebcd\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" Sep 11 11:34:42.531761 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.531733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" Sep 11 11:34:42.819906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:42.819835 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62"] Sep 11 11:34:42.823105 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:42.823079 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea7ab92_bea1_4209_bd83_490414b5ebcd.slice/crio-b4e3565d838073ddf671beb54969b6238173c9e7d6d2f139d5e0e57aaa6c267f WatchSource:0}: Error finding container b4e3565d838073ddf671beb54969b6238173c9e7d6d2f139d5e0e57aaa6c267f: Status 404 returned error can't find the container with id b4e3565d838073ddf671beb54969b6238173c9e7d6d2f139d5e0e57aaa6c267f Sep 11 11:34:43.063204 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.063175 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:43.158223 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.158191 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6dx8n"] Sep 11 11:34:43.190898 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.190870 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6dx8n"] Sep 11 11:34:43.191079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.190985 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.193309 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.193288 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9rhw9\"" Sep 11 11:34:43.193309 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.193300 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 11:34:43.193695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.193679 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 11:34:43.292939 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.292915 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:34:43.293446 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.293421 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/0.log" Sep 11 11:34:43.293559 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.293468 2580 generic.go:356] "Generic (PLEG): container finished" podID="df17a3aa-2361-405d-8165-4f5dd30e4318" containerID="1bc1fa2e781700c4323c651633344c6c18d913cf3b26be47b2f0574e62e16f97" exitCode=255 Sep 11 11:34:43.293625 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.293582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" event={"ID":"df17a3aa-2361-405d-8165-4f5dd30e4318","Type":"ContainerDied","Data":"1bc1fa2e781700c4323c651633344c6c18d913cf3b26be47b2f0574e62e16f97"} Sep 11 11:34:43.293678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.293634 2580 scope.go:117] "RemoveContainer" containerID="34afd0b2117f29191d83217a1e01e84b35913ffb5cf15b0e6c5f9108df4f3608" Sep 11 11:34:43.293793 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.293775 2580 scope.go:117] "RemoveContainer" containerID="1bc1fa2e781700c4323c651633344c6c18d913cf3b26be47b2f0574e62e16f97" Sep 11 11:34:43.294216 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.293985 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-ldgwv_openshift-console-operator(df17a3aa-2361-405d-8165-4f5dd30e4318)\"" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" podUID="df17a3aa-2361-405d-8165-4f5dd30e4318" Sep 11 11:34:43.296985 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.296132 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" event={"ID":"fea7ab92-bea1-4209-bd83-490414b5ebcd","Type":"ContainerStarted","Data":"b4e3565d838073ddf671beb54969b6238173c9e7d6d2f139d5e0e57aaa6c267f"} Sep 11 11:34:43.329773 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.329743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.329904 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.329805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.329973 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.329895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea9b246f-988f-4e91-8380-14ddd8a51ce4-data-volume\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.329973 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.329926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvf2\" (UniqueName: \"kubernetes.io/projected/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-api-access-gvvf2\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.330073 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.329981 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea9b246f-988f-4e91-8380-14ddd8a51ce4-crio-socket\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.430740 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.430652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.430916 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.430775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.430916 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.430808 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:34:43.430916 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.430843 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:34:43.430916 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.430885 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:43.930864321 +0000 UTC m=+45.565457453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : secret "insights-runtime-extractor-tls" not found Sep 11 11:34:43.430916 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.430903 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:43.930893144 +0000 UTC m=+45.565486264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : configmap "kube-rbac-proxy" not found Sep 11 11:34:43.431209 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.431024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea9b246f-988f-4e91-8380-14ddd8a51ce4-data-volume\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.431209 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.431084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvf2\" (UniqueName: \"kubernetes.io/projected/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-api-access-gvvf2\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.431209 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.431154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea9b246f-988f-4e91-8380-14ddd8a51ce4-crio-socket\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.431306 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.431213 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea9b246f-988f-4e91-8380-14ddd8a51ce4-crio-socket\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.431383 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.431364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea9b246f-988f-4e91-8380-14ddd8a51ce4-data-volume\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.441382 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.441351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvf2\" (UniqueName: \"kubernetes.io/projected/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-api-access-gvvf2\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.935914 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.935881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.936158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:43.935948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:43.936158 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.936065 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:34:43.936158 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.936081 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:34:43.936158 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.936148 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:44.936127948 +0000 UTC m=+46.570721073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : configmap "kube-rbac-proxy" not found Sep 11 11:34:43.936158 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:43.936161 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:44.936155577 +0000 UTC m=+46.570748688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : secret "insights-runtime-extractor-tls" not found Sep 11 11:34:44.137675 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:44.137637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:44.137826 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:44.137796 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 11:34:44.137896 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:44.137885 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert podName:fa5874ab-3b5c-4d44-8a50-ea744be7834d nodeName:}" failed. No retries permitted until 2026-09-11 11:34:52.137867757 +0000 UTC m=+53.772460872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7dsg8" (UID: "fa5874ab-3b5c-4d44-8a50-ea744be7834d") : secret "networking-console-plugin-cert" not found Sep 11 11:34:44.301315 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:44.301295 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:34:44.301709 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:44.301690 2580 scope.go:117] "RemoveContainer" containerID="1bc1fa2e781700c4323c651633344c6c18d913cf3b26be47b2f0574e62e16f97" Sep 11 11:34:44.301868 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:44.301847 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-ldgwv_openshift-console-operator(df17a3aa-2361-405d-8165-4f5dd30e4318)\"" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" podUID="df17a3aa-2361-405d-8165-4f5dd30e4318" Sep 11 11:34:44.945912 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:44.945622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:44.945912 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:44.945697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:44.945912 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:44.945788 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:34:44.945912 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:44.945805 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:34:44.945912 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:44.945857 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:46.945838553 +0000 UTC m=+48.580431666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : configmap "kube-rbac-proxy" not found Sep 11 11:34:44.945912 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:44.945880 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:46.945867559 +0000 UTC m=+48.580460674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : secret "insights-runtime-extractor-tls" not found Sep 11 11:34:45.124125 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:45.124098 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr9xg_543efb58-4920-4bba-bb4b-a3a4411fa7a4/dns-node-resolver/0.log" Sep 11 11:34:45.311069 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:45.310972 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" event={"ID":"fea7ab92-bea1-4209-bd83-490414b5ebcd","Type":"ContainerStarted","Data":"f4bf78e310c16e02706079f6057a052a34d22e05f03478e42b9afa25beee3dc5"} Sep 11 11:34:45.311424 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:45.311071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" event={"ID":"fea7ab92-bea1-4209-bd83-490414b5ebcd","Type":"ContainerStarted","Data":"77078e4c7159e32e65354ef38dbb9133adbf9d9bb4448d43b09efa7f88fa98d4"} Sep 11 11:34:45.331337 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:45.331281 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rpx62" podStartSLOduration=1.857478311 podStartE2EDuration="3.331264054s" totalImagesPullingTime="1.473785743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:34:45.33030604 +0000 UTC m=+46.964899184" watchObservedRunningTime="2026-09-11 11:34:45.331264054 +0000 UTC m=+46.965857252" Sep 11 11:34:46.323558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:46.323529 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vskzf_4ea77448-b67e-44e0-a935-9d6c03745028/node-ca/0.log" Sep 11 11:34:46.728998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:46.728962 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2gldc_033fa8a3-2baf-401c-9792-759c21d55ea0/serve-healthcheck-canary/0.log" Sep 11 11:34:46.924606 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:46.924566 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rpx62_fea7ab92-bea1-4209-bd83-490414b5ebcd/migrator/0.log" Sep 11 11:34:46.966306 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:46.966269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:46.966466 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:46.966399 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 11:34:46.966466 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:46.966432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:46.966567 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:46.966465 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:50.966448859 +0000 UTC m=+52.601041970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : configmap "kube-rbac-proxy" not found Sep 11 11:34:46.966567 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:46.966544 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:34:46.966665 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:46.966606 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:50.966588498 +0000 UTC m=+52.601181631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : secret "insights-runtime-extractor-tls" not found Sep 11 11:34:47.124801 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:47.124718 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rpx62_fea7ab92-bea1-4209-bd83-490414b5ebcd/graceful-termination/0.log" Sep 11 11:34:47.326619 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:47.326565 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-rxcpr_e5ecb8de-47a3-4c7a-90c8-099eb148851c/kube-storage-version-migrator-operator/0.log" Sep 11 11:34:48.581163 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:48.581127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:48.581529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:48.581200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:34:48.581529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:48.581254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:34:48.581529 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.581309 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.58128928 +0000 UTC m=+66.215882412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : configmap references non-existent config key: service-ca.crt Sep 11 11:34:48.581529 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.581340 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 11:34:48.581529 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.581340 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:48.581529 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.581386 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs podName:064268d3-1d9f-48d1-8c10-da03ed41b29c nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.581370811 +0000 UTC m=+66.215963943 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs") pod "router-default-6fcb677dbb-2nz2z" (UID: "064268d3-1d9f-48d1-8c10-da03ed41b29c") : secret "router-metrics-certs-default" not found Sep 11 11:34:48.581529 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.581413 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls podName:06d8735e-012c-40ae-99db-84fba64b0bc2 nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.58139938 +0000 UTC m=+66.215992515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zld2w" (UID: "06d8735e-012c-40ae-99db-84fba64b0bc2") : secret "cluster-monitoring-operator-tls" not found Sep 11 11:34:48.682364 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:48.682329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:34:48.682521 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:48.682413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:34:48.682521 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.682485 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 11:34:48.682521 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.682504 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f854f6d-vb2gm: secret "image-registry-tls" not found Sep 11 11:34:48.682620 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.682543 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 11:34:48.682620 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.682554 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls podName:80bf58f5-0918-4934-a5f2-f734cb77b81f nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.682540577 +0000 UTC m=+66.317133692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls") pod "image-registry-84f854f6d-vb2gm" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f") : secret "image-registry-tls" not found Sep 11 11:34:48.682620 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.682586 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls podName:4745913e-1dcb-4b94-a93d-99069e5b4182 nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.682572927 +0000 UTC m=+66.317166038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wszbv" (UID: "4745913e-1dcb-4b94-a93d-99069e5b4182") : secret "samples-operator-tls" not found Sep 11 11:34:48.783180 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:48.783146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:34:48.783344 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.783296 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 11:34:48.783383 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:48.783354 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls podName:40526ae2-253a-434c-bff9-5508117eb6c0 nodeName:}" failed. No retries permitted until 2026-09-11 11:35:04.783340474 +0000 UTC m=+66.417933585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls") pod "dns-default-jd48q" (UID: "40526ae2-253a-434c-bff9-5508117eb6c0") : secret "dns-default-metrics-tls" not found Sep 11 11:34:51.004554 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:51.004516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:51.005154 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:51.004582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:51.005154 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:51.004721 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 11:34:51.005154 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:51.004807 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls podName:ea9b246f-988f-4e91-8380-14ddd8a51ce4 nodeName:}" failed. No retries permitted until 2026-09-11 11:34:59.004783671 +0000 UTC m=+60.639376801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dx8n" (UID: "ea9b246f-988f-4e91-8380-14ddd8a51ce4") : secret "insights-runtime-extractor-tls" not found Sep 11 11:34:51.005154 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:51.005022 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea9b246f-988f-4e91-8380-14ddd8a51ce4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:51.264157 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:51.264073 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:51.264461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:51.264449 2580 scope.go:117] "RemoveContainer" containerID="1bc1fa2e781700c4323c651633344c6c18d913cf3b26be47b2f0574e62e16f97" Sep 11 11:34:51.264653 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:51.264633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-ldgwv_openshift-console-operator(df17a3aa-2361-405d-8165-4f5dd30e4318)\"" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" podUID="df17a3aa-2361-405d-8165-4f5dd30e4318" Sep 11 11:34:52.214605 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:52.214567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:34:52.215018 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:52.214708 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 11:34:52.215018 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:34:52.214808 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert podName:fa5874ab-3b5c-4d44-8a50-ea744be7834d nodeName:}" failed. No retries permitted until 2026-09-11 11:35:08.214790861 +0000 UTC m=+69.849383972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-7dsg8" (UID: "fa5874ab-3b5c-4d44-8a50-ea744be7834d") : secret "networking-console-plugin-cert" not found Sep 11 11:34:53.062976 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:53.062931 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:53.063304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:53.063291 2580 scope.go:117] "RemoveContainer" containerID="1bc1fa2e781700c4323c651633344c6c18d913cf3b26be47b2f0574e62e16f97" Sep 11 11:34:53.339022 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:53.338950 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:34:53.339022 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:53.339003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" event={"ID":"df17a3aa-2361-405d-8165-4f5dd30e4318","Type":"ContainerStarted","Data":"45806880bddc886fe7670c1106a34fa8893cc120afc7a2bd4a9da01933712960"} Sep 11 11:34:53.339510 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:53.339274 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:53.453925 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:53.453895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-ldgwv" Sep 11 11:34:59.070925 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:59.070884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:59.073465 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:59.073435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea9b246f-988f-4e91-8380-14ddd8a51ce4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dx8n\" (UID: \"ea9b246f-988f-4e91-8380-14ddd8a51ce4\") " pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:59.102568 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:59.102540 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9rhw9\"" Sep 11 11:34:59.111340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:59.111312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6dx8n" Sep 11 11:34:59.201519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:59.201497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j94vg" Sep 11 11:34:59.382851 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:34:59.382777 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6dx8n"] Sep 11 11:34:59.385911 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:34:59.385883 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9b246f_988f_4e91_8380_14ddd8a51ce4.slice/crio-41b40ed4cf74c580d3c227aa881971ede254fa42a085e9a618015e957e738a87 WatchSource:0}: Error finding container 41b40ed4cf74c580d3c227aa881971ede254fa42a085e9a618015e957e738a87: Status 404 returned error can't find the container with id 41b40ed4cf74c580d3c227aa881971ede254fa42a085e9a618015e957e738a87 Sep 11 11:35:00.362005 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:00.361924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dx8n" event={"ID":"ea9b246f-988f-4e91-8380-14ddd8a51ce4","Type":"ContainerStarted","Data":"39f4127df604d2b3feae8c78bd3fa793c542e9fe09ff50fed9019ebdc6c1579d"} Sep 11 11:35:00.362005 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:00.361974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dx8n" event={"ID":"ea9b246f-988f-4e91-8380-14ddd8a51ce4","Type":"ContainerStarted","Data":"ea945337c3a673e5fe326d050f8895094270523663f5a8864b4078d032313e3b"} Sep 11 11:35:00.362005 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:00.361994 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dx8n" event={"ID":"ea9b246f-988f-4e91-8380-14ddd8a51ce4","Type":"ContainerStarted","Data":"41b40ed4cf74c580d3c227aa881971ede254fa42a085e9a618015e957e738a87"} Sep 11 11:35:02.370627 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:02.370587 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dx8n" event={"ID":"ea9b246f-988f-4e91-8380-14ddd8a51ce4","Type":"ContainerStarted","Data":"ab964201fffb710a8dc43c0aaa150c5dd2853a82ed9f8254375f6dd9448a27fc"} Sep 11 11:35:02.388544 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:02.388489 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6dx8n" podStartSLOduration=17.098729539 podStartE2EDuration="19.388470777s" totalImagesPullingTime="2.289741238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:34:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:02.38754244 +0000 UTC m=+64.022135600" watchObservedRunningTime="2026-09-11 11:35:02.388470777 +0000 UTC m=+64.023063910" Sep 11 11:35:04.622075 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.622016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:35:04.622075 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.622079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:04.622575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.622222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:04.622575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.622254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:35:04.622575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.622303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:35:04.623017 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.622999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/064268d3-1d9f-48d1-8c10-da03ed41b29c-service-ca-bundle\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:04.624528 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.624506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/064268d3-1d9f-48d1-8c10-da03ed41b29c-metrics-certs\") pod \"router-default-6fcb677dbb-2nz2z\" (UID: \"064268d3-1d9f-48d1-8c10-da03ed41b29c\") " pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:04.624681 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.624665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/06d8735e-012c-40ae-99db-84fba64b0bc2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zld2w\" (UID: \"06d8735e-012c-40ae-99db-84fba64b0bc2\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:35:04.628909 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.628886 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 11:35:04.629089 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.629068 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 11:35:04.634856 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.634835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df5f2cf-051f-4ab9-a41d-7f0305cc486f-metrics-certs\") pod \"network-metrics-daemon-s57lp\" (UID: \"6df5f2cf-051f-4ab9-a41d-7f0305cc486f\") " pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:35:04.634965 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.634946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ebef619d-9d2e-4a27-9b1c-300daad95756-original-pull-secret\") pod \"global-pull-secret-syncer-7nn5q\" (UID: \"ebef619d-9d2e-4a27-9b1c-300daad95756\") " pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:35:04.693327 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.693295 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tk98p\"" Sep 11 11:35:04.697291 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.697262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nn5q" Sep 11 11:35:04.701743 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.701729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s57lp" Sep 11 11:35:04.723156 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.723116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:35:04.723296 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.723228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:35:04.723296 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.723285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:35:04.725526 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.725502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"image-registry-84f854f6d-vb2gm\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:35:04.726312 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.726289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4745913e-1dcb-4b94-a93d-99069e5b4182-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wszbv\" (UID: \"4745913e-1dcb-4b94-a93d-99069e5b4182\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:35:04.726473 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.726452 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln89l\" (UniqueName: \"kubernetes.io/projected/3d689159-7ed7-4c57-af29-1b64ac0491b6-kube-api-access-ln89l\") pod \"network-check-target-8pb6s\" (UID: \"3d689159-7ed7-4c57-af29-1b64ac0491b6\") " pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:35:04.823654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.823596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:35:04.826118 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.826098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40526ae2-253a-434c-bff9-5508117eb6c0-metrics-tls\") pod \"dns-default-jd48q\" (UID: \"40526ae2-253a-434c-bff9-5508117eb6c0\") " pod="openshift-dns/dns-default-jd48q" Sep 11 11:35:04.828367 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.828337 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-2clhp\"" Sep 11 11:35:04.837156 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.837138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" Sep 11 11:35:04.845154 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.845137 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p6ddf\"" Sep 11 11:35:04.853266 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.853241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:04.922181 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.922148 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mms9n\"" Sep 11 11:35:04.930435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.930246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:35:04.970017 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.969983 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v7n5d\"" Sep 11 11:35:04.977582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.977554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" Sep 11 11:35:04.981873 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.981068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7nn5q"] Sep 11 11:35:04.984060 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:04.983892 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s57lp"] Sep 11 11:35:04.987977 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:04.987946 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df5f2cf_051f_4ab9_a41d_7f0305cc486f.slice/crio-e365744f853a4abdd3fe7efaef57e49b874f65ec642c88af07e3d3fa556c2324 WatchSource:0}: Error finding container e365744f853a4abdd3fe7efaef57e49b874f65ec642c88af07e3d3fa556c2324: Status 404 returned error can't find the container with id e365744f853a4abdd3fe7efaef57e49b874f65ec642c88af07e3d3fa556c2324 Sep 11 11:35:04.988611 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:04.988494 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebef619d_9d2e_4a27_9b1c_300daad95756.slice/crio-ed226c9a16d52e037807b35cb04019ad9d8dcc9ab2364aac7d7e90b77603c81b WatchSource:0}: Error finding container ed226c9a16d52e037807b35cb04019ad9d8dcc9ab2364aac7d7e90b77603c81b: Status 404 returned error can't find the container with id ed226c9a16d52e037807b35cb04019ad9d8dcc9ab2364aac7d7e90b77603c81b Sep 11 11:35:05.005543 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.005517 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b45p4\"" Sep 11 11:35:05.014346 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.014322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:35:05.025631 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.025613 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nnmkw\"" Sep 11 11:35:05.033689 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.033656 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jd48q" Sep 11 11:35:05.115890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.115850 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w"] Sep 11 11:35:05.120559 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:05.120529 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d8735e_012c_40ae_99db_84fba64b0bc2.slice/crio-547d55fe4edf9ad6b14357053550bc9f2272ade31c514f263aeae9bb84d2ed7d WatchSource:0}: Error finding container 547d55fe4edf9ad6b14357053550bc9f2272ade31c514f263aeae9bb84d2ed7d: Status 404 returned error can't find the container with id 547d55fe4edf9ad6b14357053550bc9f2272ade31c514f263aeae9bb84d2ed7d Sep 11 11:35:05.135695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.135672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fcb677dbb-2nz2z"] Sep 11 11:35:05.138762 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:05.138714 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064268d3_1d9f_48d1_8c10_da03ed41b29c.slice/crio-4800d81ed089f6ac62a2b9401b0103f830cd12fdd02d784c8277e21345869060 WatchSource:0}: Error finding container 4800d81ed089f6ac62a2b9401b0103f830cd12fdd02d784c8277e21345869060: Status 404 returned error can't find the container with id 4800d81ed089f6ac62a2b9401b0103f830cd12fdd02d784c8277e21345869060 Sep 11 11:35:05.225896 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.225863 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f854f6d-vb2gm"] Sep 11 11:35:05.233683 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:05.233654 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80bf58f5_0918_4934_a5f2_f734cb77b81f.slice/crio-fd282414c73ca0e7ded46bd3e48a984c9a68808384a569a3208ac171cca840a0 WatchSource:0}: Error finding container fd282414c73ca0e7ded46bd3e48a984c9a68808384a569a3208ac171cca840a0: Status 404 returned error can't find the container with id fd282414c73ca0e7ded46bd3e48a984c9a68808384a569a3208ac171cca840a0 Sep 11 11:35:05.268824 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.268734 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv"] Sep 11 11:35:05.299071 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.299021 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8pb6s"] Sep 11 11:35:05.303087 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:05.303058 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d689159_7ed7_4c57_af29_1b64ac0491b6.slice/crio-0d688fb6c1c0fecb3956c6c670ff8ddb63443350598cb9ba6ce5683bcd034ba8 WatchSource:0}: Error finding container 0d688fb6c1c0fecb3956c6c670ff8ddb63443350598cb9ba6ce5683bcd034ba8: Status 404 returned error can't find the container with id 0d688fb6c1c0fecb3956c6c670ff8ddb63443350598cb9ba6ce5683bcd034ba8 Sep 11 11:35:05.320756 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.320732 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jd48q"] Sep 11 11:35:05.330450 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:05.330423 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40526ae2_253a_434c_bff9_5508117eb6c0.slice/crio-6ee8c777a3c93c091fe09a46a31fed17ce3f83637b9e8f9488f15ec6f82e28ce WatchSource:0}: Error finding container 6ee8c777a3c93c091fe09a46a31fed17ce3f83637b9e8f9488f15ec6f82e28ce: Status 404 returned error can't find the container with id 6ee8c777a3c93c091fe09a46a31fed17ce3f83637b9e8f9488f15ec6f82e28ce Sep 11 11:35:05.383623 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.383590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jd48q" event={"ID":"40526ae2-253a-434c-bff9-5508117eb6c0","Type":"ContainerStarted","Data":"6ee8c777a3c93c091fe09a46a31fed17ce3f83637b9e8f9488f15ec6f82e28ce"} Sep 11 11:35:05.385281 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.385252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8pb6s" event={"ID":"3d689159-7ed7-4c57-af29-1b64ac0491b6","Type":"ContainerStarted","Data":"c9c78de8dc9aeb00cb506cc4f70dc45798644b83e0927392b8e1dc5d49bcc8a6"} Sep 11 11:35:05.385374 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.385294 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8pb6s" event={"ID":"3d689159-7ed7-4c57-af29-1b64ac0491b6","Type":"ContainerStarted","Data":"0d688fb6c1c0fecb3956c6c670ff8ddb63443350598cb9ba6ce5683bcd034ba8"} Sep 11 11:35:05.385440 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.385429 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:35:05.386549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.386527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" event={"ID":"4745913e-1dcb-4b94-a93d-99069e5b4182","Type":"ContainerStarted","Data":"ee641db18bf0c642d556e4f2855452dcdb7cd14d443c17e734636cef5dabc2c3"} Sep 11 11:35:05.388082 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.388059 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" event={"ID":"80bf58f5-0918-4934-a5f2-f734cb77b81f","Type":"ContainerStarted","Data":"1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e"} Sep 11 11:35:05.388212 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.388106 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:35:05.388212 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.388125 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" event={"ID":"80bf58f5-0918-4934-a5f2-f734cb77b81f","Type":"ContainerStarted","Data":"fd282414c73ca0e7ded46bd3e48a984c9a68808384a569a3208ac171cca840a0"} Sep 11 11:35:05.389472 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.389421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" event={"ID":"06d8735e-012c-40ae-99db-84fba64b0bc2","Type":"ContainerStarted","Data":"547d55fe4edf9ad6b14357053550bc9f2272ade31c514f263aeae9bb84d2ed7d"} Sep 11 11:35:05.390963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.390935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" event={"ID":"064268d3-1d9f-48d1-8c10-da03ed41b29c","Type":"ContainerStarted","Data":"8783142b03e33d6ac17444b1118d79f83eed14bd1023d632ae6c8552c5a36fd9"} Sep 11 11:35:05.391057 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.390970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" event={"ID":"064268d3-1d9f-48d1-8c10-da03ed41b29c","Type":"ContainerStarted","Data":"4800d81ed089f6ac62a2b9401b0103f830cd12fdd02d784c8277e21345869060"} Sep 11 11:35:05.392214 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.392188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s57lp" event={"ID":"6df5f2cf-051f-4ab9-a41d-7f0305cc486f","Type":"ContainerStarted","Data":"e365744f853a4abdd3fe7efaef57e49b874f65ec642c88af07e3d3fa556c2324"} Sep 11 11:35:05.393253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.393235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7nn5q" event={"ID":"ebef619d-9d2e-4a27-9b1c-300daad95756","Type":"ContainerStarted","Data":"ed226c9a16d52e037807b35cb04019ad9d8dcc9ab2364aac7d7e90b77603c81b"} Sep 11 11:35:05.404235 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.404183 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8pb6s" podStartSLOduration=66.404166396 podStartE2EDuration="1m6.404166396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:05.403876574 +0000 UTC m=+67.038469707" watchObservedRunningTime="2026-09-11 11:35:05.404166396 +0000 UTC m=+67.038759528" Sep 11 11:35:05.425556 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.425117 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" podStartSLOduration=56.425097772 podStartE2EDuration="56.425097772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:05.424558504 +0000 UTC m=+67.059151629" watchObservedRunningTime="2026-09-11 11:35:05.425097772 +0000 UTC m=+67.059690907" Sep 11 11:35:05.853672 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.853629 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:05.856749 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.856715 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:05.879537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:05.879467 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" podStartSLOduration=66.879441539 podStartE2EDuration="1m6.879441539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:05.448497112 +0000 UTC m=+67.083090282" watchObservedRunningTime="2026-09-11 11:35:05.879441539 +0000 UTC m=+67.514034671" Sep 11 11:35:06.402342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:06.401465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s57lp" event={"ID":"6df5f2cf-051f-4ab9-a41d-7f0305cc486f","Type":"ContainerStarted","Data":"23aec5bc329c8077340c4d3ed166e0207340ca907737743aacd4694469c722fd"} Sep 11 11:35:06.402342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:06.401519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s57lp" event={"ID":"6df5f2cf-051f-4ab9-a41d-7f0305cc486f","Type":"ContainerStarted","Data":"d3bcbde49b83908e09bd7f0b21d9f5a968852353be38082b4a4de3dff04aeff7"} Sep 11 11:35:06.402342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:06.402173 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:06.404197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:06.404156 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6fcb677dbb-2nz2z" Sep 11 11:35:06.417822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:06.417765 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s57lp" podStartSLOduration=66.354675692 podStartE2EDuration="1m7.417744294s" totalImagesPullingTime="1.063068602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:06.416974008 +0000 UTC m=+68.051567142" watchObservedRunningTime="2026-09-11 11:35:06.417744294 +0000 UTC m=+68.052337428" Sep 11 11:35:08.257757 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.257717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:35:08.261487 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.261455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fa5874ab-3b5c-4d44-8a50-ea744be7834d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-7dsg8\" (UID: \"fa5874ab-3b5c-4d44-8a50-ea744be7834d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:35:08.291880 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.291843 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-qvl9x"] Sep 11 11:35:08.294658 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.294635 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qvl9x" Sep 11 11:35:08.300497 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.300469 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cg6qj\"" Sep 11 11:35:08.300666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.300604 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 11:35:08.301111 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.300937 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 11:35:08.313603 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.313548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qvl9x"] Sep 11 11:35:08.358828 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.358781 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmf6d\" (UniqueName: \"kubernetes.io/projected/3609dc30-fcab-4fb8-bc9a-033e50ef6a3e-kube-api-access-cmf6d\") pod \"downloads-76f99584f4-qvl9x\" (UID: \"3609dc30-fcab-4fb8-bc9a-033e50ef6a3e\") " pod="openshift-console/downloads-76f99584f4-qvl9x" Sep 11 11:35:08.422412 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.422390 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-gwc9z\"" Sep 11 11:35:08.430868 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.430846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" Sep 11 11:35:08.460049 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.459952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmf6d\" (UniqueName: \"kubernetes.io/projected/3609dc30-fcab-4fb8-bc9a-033e50ef6a3e-kube-api-access-cmf6d\") pod \"downloads-76f99584f4-qvl9x\" (UID: \"3609dc30-fcab-4fb8-bc9a-033e50ef6a3e\") " pod="openshift-console/downloads-76f99584f4-qvl9x" Sep 11 11:35:08.474805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.474742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmf6d\" (UniqueName: \"kubernetes.io/projected/3609dc30-fcab-4fb8-bc9a-033e50ef6a3e-kube-api-access-cmf6d\") pod \"downloads-76f99584f4-qvl9x\" (UID: \"3609dc30-fcab-4fb8-bc9a-033e50ef6a3e\") " pod="openshift-console/downloads-76f99584f4-qvl9x" Sep 11 11:35:08.608263 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.608228 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qvl9x" Sep 11 11:35:08.963626 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.963593 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n"] Sep 11 11:35:08.965900 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.965881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:08.968138 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.968109 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 11:35:08.968138 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.968131 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-btqhh\"" Sep 11 11:35:08.976692 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:08.976663 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n"] Sep 11 11:35:09.064494 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.064449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5859eafb-5526-4708-8354-06c3f247d6f1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vlj5n\" (UID: \"5859eafb-5526-4708-8354-06c3f247d6f1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:09.165390 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.165350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5859eafb-5526-4708-8354-06c3f247d6f1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vlj5n\" (UID: \"5859eafb-5526-4708-8354-06c3f247d6f1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:09.165560 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:35:09.165530 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 11 11:35:09.165617 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:35:09.165611 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5859eafb-5526-4708-8354-06c3f247d6f1-tls-certificates podName:5859eafb-5526-4708-8354-06c3f247d6f1 nodeName:}" failed. No retries permitted until 2026-09-11 11:35:09.66559002 +0000 UTC m=+71.300183134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/5859eafb-5526-4708-8354-06c3f247d6f1-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-vlj5n" (UID: "5859eafb-5526-4708-8354-06c3f247d6f1") : secret "prometheus-operator-admission-webhook-tls" not found Sep 11 11:35:09.415559 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.415525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" event={"ID":"4745913e-1dcb-4b94-a93d-99069e5b4182","Type":"ContainerStarted","Data":"77f6127ad8e22f547f711f6a11828c8714481a5b8ccc8aaa7feeeb96b627d4c3"} Sep 11 11:35:09.417318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.417292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" event={"ID":"06d8735e-012c-40ae-99db-84fba64b0bc2","Type":"ContainerStarted","Data":"1da3f2cd8fb22710e9c2f6b479e38634177b15cb19381982b51ee105f84ed0f4"} Sep 11 11:35:09.434487 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.434136 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zld2w" podStartSLOduration=57.140639763 podStartE2EDuration="1m0.434117361s" totalImagesPullingTime="3.293477598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:09.433540622 +0000 UTC m=+71.068133755" watchObservedRunningTime="2026-09-11 11:35:09.434117361 +0000 UTC m=+71.068710533" Sep 11 11:35:09.669636 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.669581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5859eafb-5526-4708-8354-06c3f247d6f1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vlj5n\" (UID: \"5859eafb-5526-4708-8354-06c3f247d6f1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:09.674460 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.673160 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5859eafb-5526-4708-8354-06c3f247d6f1-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-vlj5n\" (UID: \"5859eafb-5526-4708-8354-06c3f247d6f1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:09.880119 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:09.880077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:10.264854 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.264785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8"] Sep 11 11:35:10.278094 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:10.278052 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5874ab_3b5c_4d44_8a50_ea744be7834d.slice/crio-5b6ebdcbff273c0cdaf09ffa58f82f7a003df24931a0d26abf3e8f49b3e0ba57 WatchSource:0}: Error finding container 5b6ebdcbff273c0cdaf09ffa58f82f7a003df24931a0d26abf3e8f49b3e0ba57: Status 404 returned error can't find the container with id 5b6ebdcbff273c0cdaf09ffa58f82f7a003df24931a0d26abf3e8f49b3e0ba57 Sep 11 11:35:10.300122 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.300090 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qvl9x"] Sep 11 11:35:10.301451 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.301366 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n"] Sep 11 11:35:10.304921 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:10.304892 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5859eafb_5526_4708_8354_06c3f247d6f1.slice/crio-ece5debce26b1707ea04ae0d9ac757d615dc9115c72d43c7f3d106f3d9116eb7 WatchSource:0}: Error finding container ece5debce26b1707ea04ae0d9ac757d615dc9115c72d43c7f3d106f3d9116eb7: Status 404 returned error can't find the container with id ece5debce26b1707ea04ae0d9ac757d615dc9115c72d43c7f3d106f3d9116eb7 Sep 11 11:35:10.424579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.424537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qvl9x" event={"ID":"3609dc30-fcab-4fb8-bc9a-033e50ef6a3e","Type":"ContainerStarted","Data":"f3cd523b56334b0e9745f2ac9ffe3e88d4f25cffb9b2231dcda7f87a7580e9c3"} Sep 11 11:35:10.428009 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.427949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jd48q" event={"ID":"40526ae2-253a-434c-bff9-5508117eb6c0","Type":"ContainerStarted","Data":"ca344c4c05929539ebd46d380ef056cfaf5cc6b08da0a25f37b038fbc04a93c7"} Sep 11 11:35:10.428009 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.428000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jd48q" event={"ID":"40526ae2-253a-434c-bff9-5508117eb6c0","Type":"ContainerStarted","Data":"f7ead3080196d7c98fff4803547900fa890bf3c217eb89f9743fc4b21bf90efe"} Sep 11 11:35:10.428248 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.428072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jd48q" Sep 11 11:35:10.430161 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.430137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" event={"ID":"4745913e-1dcb-4b94-a93d-99069e5b4182","Type":"ContainerStarted","Data":"9d5a1ed21cbedf51198c9d964f46f23e5a7f1cb82e3e18a840caf6f4382586aa"} Sep 11 11:35:10.431642 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.431612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" event={"ID":"5859eafb-5526-4708-8354-06c3f247d6f1","Type":"ContainerStarted","Data":"ece5debce26b1707ea04ae0d9ac757d615dc9115c72d43c7f3d106f3d9116eb7"} Sep 11 11:35:10.433393 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.433371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" event={"ID":"fa5874ab-3b5c-4d44-8a50-ea744be7834d","Type":"ContainerStarted","Data":"5b6ebdcbff273c0cdaf09ffa58f82f7a003df24931a0d26abf3e8f49b3e0ba57"} Sep 11 11:35:10.445405 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.445357 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jd48q" podStartSLOduration=35.35514075 podStartE2EDuration="38.445341742s" totalImagesPullingTime="3.090200992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:10.444266268 +0000 UTC m=+72.078859415" watchObservedRunningTime="2026-09-11 11:35:10.445341742 +0000 UTC m=+72.079934876" Sep 11 11:35:10.463732 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:10.463685 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wszbv" podStartSLOduration=48.356680857 podStartE2EDuration="51.463664923s" totalImagesPullingTime="3.106984066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:10.462395484 +0000 UTC m=+72.096988619" watchObservedRunningTime="2026-09-11 11:35:10.463664923 +0000 UTC m=+72.098258060" Sep 11 11:35:11.444759 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:11.444668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7nn5q" event={"ID":"ebef619d-9d2e-4a27-9b1c-300daad95756","Type":"ContainerStarted","Data":"8ad563fcd7d0688b24b4411d03f0609719e4bfb74a395fe272516cf8f448e7a3"} Sep 11 11:35:11.461402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:11.461346 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7nn5q" podStartSLOduration=67.144371866 podStartE2EDuration="1m12.461314066s" totalImagesPullingTime="5.3169422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 11:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:11.460664032 +0000 UTC m=+73.095257166" watchObservedRunningTime="2026-09-11 11:35:11.461314066 +0000 UTC m=+73.095907199" Sep 11 11:35:13.457143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:13.457099 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" event={"ID":"fa5874ab-3b5c-4d44-8a50-ea744be7834d","Type":"ContainerStarted","Data":"f409cb3e1e8d1fc8bbe69dad2ab3787e6fb79ef06e7b92d683cfb840fe161ef6"} Sep 11 11:35:13.460404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:13.460374 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" event={"ID":"5859eafb-5526-4708-8354-06c3f247d6f1","Type":"ContainerStarted","Data":"eb497edfdd27222b20d5f8c14b71e38417c79f7a4fce4fdd3b719f17f2197483"} Sep 11 11:35:13.460630 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:13.460606 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:13.466144 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:13.466118 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" Sep 11 11:35:13.475721 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:13.475665 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-7dsg8" podStartSLOduration=35.274734514 podStartE2EDuration="37.475641119s" totalImagesPullingTime="2.200906605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:13.473764177 +0000 UTC m=+75.108357322" watchObservedRunningTime="2026-09-11 11:35:13.475641119 +0000 UTC m=+75.110234252" Sep 11 11:35:13.489184 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:13.489122 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-vlj5n" podStartSLOduration=3.300471528 podStartE2EDuration="5.489098856s" totalImagesPullingTime="2.188627328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:13.488356806 +0000 UTC m=+75.122949944" watchObservedRunningTime="2026-09-11 11:35:13.489098856 +0000 UTC m=+75.123691990" Sep 11 11:35:20.448397 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:20.448230 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jd48q" Sep 11 11:35:22.758488 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.758454 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l9zv9"] Sep 11 11:35:22.762214 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.762180 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.764426 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.764392 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 11:35:22.767214 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.767194 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 11:35:22.767749 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.767728 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fcpz2\"" Sep 11 11:35:22.767992 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.767975 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 11:35:22.768295 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.768277 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 11:35:22.886029 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.885989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-tls\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-wtmp\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-sys\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d472299c-8964-4f5b-895a-53d87e2224a0-metrics-client-ca\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjpp\" (UniqueName: \"kubernetes.io/projected/d472299c-8964-4f5b-895a-53d87e2224a0-kube-api-access-5zjpp\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-root\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.886415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.886344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-textfile\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.987686 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-sys\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.987853 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d472299c-8964-4f5b-895a-53d87e2224a0-metrics-client-ca\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.987853 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjpp\" (UniqueName: \"kubernetes.io/projected/d472299c-8964-4f5b-895a-53d87e2224a0-kube-api-access-5zjpp\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.987853 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.987853 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-root\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.987853 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-sys\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.987853 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-textfile\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988239 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-root\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988239 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988239 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-tls\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988239 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.987941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-wtmp\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988239 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.988133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-wtmp\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988239 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:35:22.988225 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 11:35:22.988514 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:35:22.988299 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-tls podName:d472299c-8964-4f5b-895a-53d87e2224a0 nodeName:}" failed. No retries permitted until 2026-09-11 11:35:23.488277622 +0000 UTC m=+85.122870751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-tls") pod "node-exporter-l9zv9" (UID: "d472299c-8964-4f5b-895a-53d87e2224a0") : secret "node-exporter-tls" not found Sep 11 11:35:22.988514 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.988352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d472299c-8964-4f5b-895a-53d87e2224a0-metrics-client-ca\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988627 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.988551 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-textfile\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.988765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.988745 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:22.990810 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:22.990787 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:23.005531 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:23.005499 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjpp\" (UniqueName: \"kubernetes.io/projected/d472299c-8964-4f5b-895a-53d87e2224a0-kube-api-access-5zjpp\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:23.493124 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:23.493084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-tls\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:23.495959 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:23.495931 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d472299c-8964-4f5b-895a-53d87e2224a0-node-exporter-tls\") pod \"node-exporter-l9zv9\" (UID: \"d472299c-8964-4f5b-895a-53d87e2224a0\") " pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:23.673564 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:23.673531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9zv9" Sep 11 11:35:26.407069 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:26.406681 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:35:27.032373 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:35:27.032347 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd472299c_8964_4f5b_895a_53d87e2224a0.slice/crio-7881676cf7b2762afbf0438ebc943afd5d43777b1733590d38f20df63334c769 WatchSource:0}: Error finding container 7881676cf7b2762afbf0438ebc943afd5d43777b1733590d38f20df63334c769: Status 404 returned error can't find the container with id 7881676cf7b2762afbf0438ebc943afd5d43777b1733590d38f20df63334c769 Sep 11 11:35:27.535018 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:27.534977 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qvl9x" event={"ID":"3609dc30-fcab-4fb8-bc9a-033e50ef6a3e","Type":"ContainerStarted","Data":"9bc00466ccb8c174710d208696c659c7af6cf60492537fba6e2a3c04696ae48b"} Sep 11 11:35:27.536170 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:27.536104 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-qvl9x" Sep 11 11:35:27.537883 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:27.537541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9zv9" event={"ID":"d472299c-8964-4f5b-895a-53d87e2224a0","Type":"ContainerStarted","Data":"7881676cf7b2762afbf0438ebc943afd5d43777b1733590d38f20df63334c769"} Sep 11 11:35:27.542132 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:27.542113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-qvl9x" Sep 11 11:35:27.617946 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:27.617868 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-qvl9x" podStartSLOduration=2.680825638 podStartE2EDuration="19.617843825s" totalImagesPullingTime="16.937018187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:27.572457869 +0000 UTC m=+89.207051153" watchObservedRunningTime="2026-09-11 11:35:27.617843825 +0000 UTC m=+89.252436959" Sep 11 11:35:28.545006 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:28.544965 2580 generic.go:356] "Generic (PLEG): container finished" podID="d472299c-8964-4f5b-895a-53d87e2224a0" containerID="3eacc05c41738010664452753e0b8a169a89fdefccacc584f56bd07ed2f49097" exitCode=0 Sep 11 11:35:28.545508 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:28.545070 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9zv9" event={"ID":"d472299c-8964-4f5b-895a-53d87e2224a0","Type":"ContainerDied","Data":"3eacc05c41738010664452753e0b8a169a89fdefccacc584f56bd07ed2f49097"} Sep 11 11:35:29.553726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:29.553680 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9zv9" event={"ID":"d472299c-8964-4f5b-895a-53d87e2224a0","Type":"ContainerStarted","Data":"11f72fa107547377e0db4bdffbc2724609bf3e220bd802aaf6f5bb5d56481e7b"} Sep 11 11:35:29.554228 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:29.553759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9zv9" event={"ID":"d472299c-8964-4f5b-895a-53d87e2224a0","Type":"ContainerStarted","Data":"a335da8b028f68984bc1f0fa38aca3a6af40de05156d95583e2e3515988202e6"} Sep 11 11:35:29.575301 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:29.575242 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l9zv9" podStartSLOduration=6.461324726 podStartE2EDuration="7.575222668s" totalImagesPullingTime="1.028465185s" totalInitContainerRuntime="85.432757ms" isStatefulPod=true podCreationTimestamp="2026-09-11 11:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:35:29.573322137 +0000 UTC m=+91.207915272" watchObservedRunningTime="2026-09-11 11:35:29.575222668 +0000 UTC m=+91.209815805" Sep 11 11:35:30.318296 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:30.318255 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f854f6d-vb2gm"] Sep 11 11:35:36.404179 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:36.404146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8pb6s" Sep 11 11:35:46.623173 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:46.623138 2580 generic.go:356] "Generic (PLEG): container finished" podID="e5ecb8de-47a3-4c7a-90c8-099eb148851c" containerID="7f1079fbec01d60e036450a263f9aaeb1f3d9623ff16ee13371d4af5c2a52e68" exitCode=0 Sep 11 11:35:46.623534 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:46.623217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" event={"ID":"e5ecb8de-47a3-4c7a-90c8-099eb148851c","Type":"ContainerDied","Data":"7f1079fbec01d60e036450a263f9aaeb1f3d9623ff16ee13371d4af5c2a52e68"} Sep 11 11:35:46.623600 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:46.623544 2580 scope.go:117] "RemoveContainer" containerID="7f1079fbec01d60e036450a263f9aaeb1f3d9623ff16ee13371d4af5c2a52e68" Sep 11 11:35:47.628489 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:47.628460 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-rxcpr" event={"ID":"e5ecb8de-47a3-4c7a-90c8-099eb148851c","Type":"ContainerStarted","Data":"1435cd45cc4f3458b633d37a81552b0663a9c9df998107918d43eb7367ce86c1"} Sep 11 11:35:55.341143 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.341084 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" podUID="80bf58f5-0918-4934-a5f2-f734cb77b81f" containerName="registry" containerID="cri-o://1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e" gracePeriod=30 Sep 11 11:35:55.619290 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.619256 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:35:55.642880 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.642839 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-image-registry-private-configuration\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.642888 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-certificates\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.642917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80bf58f5-0918-4934-a5f2-f734cb77b81f-ca-trust-extracted\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.642942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-trusted-ca\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.642976 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.642999 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2nq5\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-kube-api-access-j2nq5\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.643052 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-installation-pull-secrets\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643357 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.643102 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-bound-sa-token\") pod \"80bf58f5-0918-4934-a5f2-f734cb77b81f\" (UID: \"80bf58f5-0918-4934-a5f2-f734cb77b81f\") " Sep 11 11:35:55.643513 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.643449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:35:55.644015 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.643981 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:35:55.646583 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.646539 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:35:55.646683 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.646606 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:35:55.646745 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.646717 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-kube-api-access-j2nq5" (OuterVolumeSpecName: "kube-api-access-j2nq5") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "kube-api-access-j2nq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:35:55.646913 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.646882 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:35:55.646957 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.646932 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:35:55.653390 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.653368 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80bf58f5-0918-4934-a5f2-f734cb77b81f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "80bf58f5-0918-4934-a5f2-f734cb77b81f" (UID: "80bf58f5-0918-4934-a5f2-f734cb77b81f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:35:55.659917 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.659891 2580 generic.go:356] "Generic (PLEG): container finished" podID="80bf58f5-0918-4934-a5f2-f734cb77b81f" containerID="1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e" exitCode=0 Sep 11 11:35:55.659994 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.659933 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" Sep 11 11:35:55.659994 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.659966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" event={"ID":"80bf58f5-0918-4934-a5f2-f734cb77b81f","Type":"ContainerDied","Data":"1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e"} Sep 11 11:35:55.660097 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.660015 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f854f6d-vb2gm" event={"ID":"80bf58f5-0918-4934-a5f2-f734cb77b81f","Type":"ContainerDied","Data":"fd282414c73ca0e7ded46bd3e48a984c9a68808384a569a3208ac171cca840a0"} Sep 11 11:35:55.660097 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.660048 2580 scope.go:117] "RemoveContainer" containerID="1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e" Sep 11 11:35:55.668581 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.668567 2580 scope.go:117] "RemoveContainer" containerID="1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e" Sep 11 11:35:55.668870 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:35:55.668852 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e\": container with ID starting with 1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e not found: ID does not exist" containerID="1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e" Sep 11 11:35:55.668922 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.668877 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e"} err="failed to get container status \"1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e\": rpc error: code = NotFound desc = could not find container \"1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e\": container with ID starting with 1bb3a7e20e304bf8e02a5d7379eeb180689434c7ec349656d98b455dcb9f953e not found: ID does not exist" Sep 11 11:35:55.680471 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.680452 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f854f6d-vb2gm"] Sep 11 11:35:55.684278 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.684260 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84f854f6d-vb2gm"] Sep 11 11:35:55.744579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744554 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-image-registry-private-configuration\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:55.744579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744576 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-certificates\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:55.744726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744586 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80bf58f5-0918-4934-a5f2-f734cb77b81f-ca-trust-extracted\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:55.744726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744595 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80bf58f5-0918-4934-a5f2-f734cb77b81f-trusted-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:55.744726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744603 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-registry-tls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:55.744726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744610 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2nq5\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-kube-api-access-j2nq5\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:55.744726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744618 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80bf58f5-0918-4934-a5f2-f734cb77b81f-installation-pull-secrets\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:55.744726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:55.744627 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80bf58f5-0918-4934-a5f2-f734cb77b81f-bound-sa-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:35:56.871317 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:35:56.871285 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80bf58f5-0918-4934-a5f2-f734cb77b81f" path="/var/lib/kubelet/pods/80bf58f5-0918-4934-a5f2-f734cb77b81f/volumes" Sep 11 11:36:02.689347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:02.689309 2580 generic.go:356] "Generic (PLEG): container finished" podID="e3a4679a-bf25-4d80-810e-4df2cf36dd98" containerID="f8d3787d6915a1f486303589d87b8fa6f4c495564c4438da8e8cea9fc7368488" exitCode=0 Sep 11 11:36:02.689753 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:02.689383 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x7dfh" event={"ID":"e3a4679a-bf25-4d80-810e-4df2cf36dd98","Type":"ContainerDied","Data":"f8d3787d6915a1f486303589d87b8fa6f4c495564c4438da8e8cea9fc7368488"} Sep 11 11:36:02.689753 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:02.689736 2580 scope.go:117] "RemoveContainer" containerID="f8d3787d6915a1f486303589d87b8fa6f4c495564c4438da8e8cea9fc7368488" Sep 11 11:36:03.694458 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:03.694426 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-x7dfh" event={"ID":"e3a4679a-bf25-4d80-810e-4df2cf36dd98","Type":"ContainerStarted","Data":"215d6767d041f715cebd8ab277a24aac444fcfd5246eebf81f4e8ed3e7c99a1f"} Sep 11 11:36:07.711159 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:07.711126 2580 generic.go:356] "Generic (PLEG): container finished" podID="ec29893b-f200-4103-8d1d-63bbe5194a1a" containerID="351c681252c475792aae90e0e4ee6173b8f141260578aec3f191533cdc0743bb" exitCode=0 Sep 11 11:36:07.711738 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:07.711209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" event={"ID":"ec29893b-f200-4103-8d1d-63bbe5194a1a","Type":"ContainerDied","Data":"351c681252c475792aae90e0e4ee6173b8f141260578aec3f191533cdc0743bb"} Sep 11 11:36:07.711738 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:07.711661 2580 scope.go:117] "RemoveContainer" containerID="351c681252c475792aae90e0e4ee6173b8f141260578aec3f191533cdc0743bb" Sep 11 11:36:08.716807 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:36:08.716772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-d9hh4" event={"ID":"ec29893b-f200-4103-8d1d-63bbe5194a1a","Type":"ContainerStarted","Data":"560b9fb6ecec6a517786bd0186ff3b37dabf4b6c72580abb865578a181e6b7d5"} Sep 11 11:38:33.807253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.807171 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2"] Sep 11 11:38:33.807691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.807454 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80bf58f5-0918-4934-a5f2-f734cb77b81f" containerName="registry" Sep 11 11:38:33.807691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.807465 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80bf58f5-0918-4934-a5f2-f734cb77b81f" containerName="registry" Sep 11 11:38:33.807691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.807519 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80bf58f5-0918-4934-a5f2-f734cb77b81f" containerName="registry" Sep 11 11:38:33.809354 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.809339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:33.812402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.812377 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 11:38:33.812521 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.812485 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4t6v\"" Sep 11 11:38:33.813005 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.812991 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 11:38:33.827670 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.827648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2"] Sep 11 11:38:33.937015 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.936986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:33.937190 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.937026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdwg\" (UniqueName: \"kubernetes.io/projected/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-kube-api-access-qqdwg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:33.937190 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:33.937128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.038236 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.038197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.038418 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.038270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.038418 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.038311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdwg\" (UniqueName: \"kubernetes.io/projected/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-kube-api-access-qqdwg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.038583 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.038562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.038705 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.038686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.046906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.046873 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdwg\" (UniqueName: \"kubernetes.io/projected/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-kube-api-access-qqdwg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.118766 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.118685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:34.389133 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:34.389049 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2"] Sep 11 11:38:34.391918 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:38:34.391891 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef91e57_e6d6_4bc4_8010_3008b785ed3b.slice/crio-d30da0c29bdd09e1401dfac511f194d6eef2501904151e0f06d71552a6089059 WatchSource:0}: Error finding container d30da0c29bdd09e1401dfac511f194d6eef2501904151e0f06d71552a6089059: Status 404 returned error can't find the container with id d30da0c29bdd09e1401dfac511f194d6eef2501904151e0f06d71552a6089059 Sep 11 11:38:35.263897 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:35.263859 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" event={"ID":"6ef91e57-e6d6-4bc4-8010-3008b785ed3b","Type":"ContainerStarted","Data":"d30da0c29bdd09e1401dfac511f194d6eef2501904151e0f06d71552a6089059"} Sep 11 11:38:41.289611 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:41.289584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" event={"ID":"6ef91e57-e6d6-4bc4-8010-3008b785ed3b","Type":"ContainerStarted","Data":"deac9e0c62fdbbac028dbb99536f032ecbb45e7b2aaab7fb6a2b051cef1b3757"} Sep 11 11:38:42.294829 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:42.294786 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerID="deac9e0c62fdbbac028dbb99536f032ecbb45e7b2aaab7fb6a2b051cef1b3757" exitCode=0 Sep 11 11:38:42.295356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:42.294840 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" event={"ID":"6ef91e57-e6d6-4bc4-8010-3008b785ed3b","Type":"ContainerDied","Data":"deac9e0c62fdbbac028dbb99536f032ecbb45e7b2aaab7fb6a2b051cef1b3757"} Sep 11 11:38:44.304411 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:44.304378 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerID="d9aecb1a5b6f65e88cf58917a4798d2fb57092f2c612c15a868414f523238f1b" exitCode=0 Sep 11 11:38:44.304977 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:44.304459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" event={"ID":"6ef91e57-e6d6-4bc4-8010-3008b785ed3b","Type":"ContainerDied","Data":"d9aecb1a5b6f65e88cf58917a4798d2fb57092f2c612c15a868414f523238f1b"} Sep 11 11:38:52.338893 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:52.338859 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerID="5498612d287a55ba2e8eb2f577a1890c695d4bc928ec10ec9406dc9949ea02a3" exitCode=0 Sep 11 11:38:52.339278 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:52.338916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" event={"ID":"6ef91e57-e6d6-4bc4-8010-3008b785ed3b","Type":"ContainerDied","Data":"5498612d287a55ba2e8eb2f577a1890c695d4bc928ec10ec9406dc9949ea02a3"} Sep 11 11:38:53.467500 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.467478 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:53.478935 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.478911 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqdwg\" (UniqueName: \"kubernetes.io/projected/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-kube-api-access-qqdwg\") pod \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " Sep 11 11:38:53.479050 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.478975 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-bundle\") pod \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " Sep 11 11:38:53.479050 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.478993 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-util\") pod \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\" (UID: \"6ef91e57-e6d6-4bc4-8010-3008b785ed3b\") " Sep 11 11:38:53.479601 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.479576 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-bundle" (OuterVolumeSpecName: "bundle") pod "6ef91e57-e6d6-4bc4-8010-3008b785ed3b" (UID: "6ef91e57-e6d6-4bc4-8010-3008b785ed3b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:38:53.481173 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.481144 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-kube-api-access-qqdwg" (OuterVolumeSpecName: "kube-api-access-qqdwg") pod "6ef91e57-e6d6-4bc4-8010-3008b785ed3b" (UID: "6ef91e57-e6d6-4bc4-8010-3008b785ed3b"). InnerVolumeSpecName "kube-api-access-qqdwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:38:53.483257 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.483216 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-util" (OuterVolumeSpecName: "util") pod "6ef91e57-e6d6-4bc4-8010-3008b785ed3b" (UID: "6ef91e57-e6d6-4bc4-8010-3008b785ed3b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:38:53.579495 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.579461 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdwg\" (UniqueName: \"kubernetes.io/projected/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-kube-api-access-qqdwg\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:38:53.579495 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.579494 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:38:53.579698 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:53.579508 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6ef91e57-e6d6-4bc4-8010-3008b785ed3b-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:38:54.347902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:54.347870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" event={"ID":"6ef91e57-e6d6-4bc4-8010-3008b785ed3b","Type":"ContainerDied","Data":"d30da0c29bdd09e1401dfac511f194d6eef2501904151e0f06d71552a6089059"} Sep 11 11:38:54.347902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:54.347893 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540zcrp2" Sep 11 11:38:54.348120 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:54.347913 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30da0c29bdd09e1401dfac511f194d6eef2501904151e0f06d71552a6089059" Sep 11 11:38:58.780745 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:58.780713 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:38:58.783773 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:58.783749 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:38:58.790965 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:38:58.790946 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 11:39:02.017247 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017206 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn"] Sep 11 11:39:02.019402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017507 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerName="extract" Sep 11 11:39:02.019402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017523 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerName="extract" Sep 11 11:39:02.019402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017533 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerName="util" Sep 11 11:39:02.019402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017539 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerName="util" Sep 11 11:39:02.019402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017550 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerName="pull" Sep 11 11:39:02.019402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017555 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerName="pull" Sep 11 11:39:02.019402 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.017620 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef91e57-e6d6-4bc4-8010-3008b785ed3b" containerName="extract" Sep 11 11:39:02.020265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.020249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.022954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.022929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tfdfj\"" Sep 11 11:39:02.023116 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.022982 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 11:39:02.023166 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.023134 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:39:02.036342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.036311 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn"] Sep 11 11:39:02.042592 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.042564 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnrm\" (UniqueName: \"kubernetes.io/projected/35af09b3-7c0b-4371-87eb-9c6a3c338d2b-kube-api-access-rpnrm\") pod \"cert-manager-operator-controller-manager-55c444bf86-zzkgn\" (UID: \"35af09b3-7c0b-4371-87eb-9c6a3c338d2b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.042731 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.042605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35af09b3-7c0b-4371-87eb-9c6a3c338d2b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zzkgn\" (UID: \"35af09b3-7c0b-4371-87eb-9c6a3c338d2b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.143135 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.143099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnrm\" (UniqueName: \"kubernetes.io/projected/35af09b3-7c0b-4371-87eb-9c6a3c338d2b-kube-api-access-rpnrm\") pod \"cert-manager-operator-controller-manager-55c444bf86-zzkgn\" (UID: \"35af09b3-7c0b-4371-87eb-9c6a3c338d2b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.143135 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.143151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35af09b3-7c0b-4371-87eb-9c6a3c338d2b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zzkgn\" (UID: \"35af09b3-7c0b-4371-87eb-9c6a3c338d2b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.143533 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.143512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35af09b3-7c0b-4371-87eb-9c6a3c338d2b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zzkgn\" (UID: \"35af09b3-7c0b-4371-87eb-9c6a3c338d2b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.156589 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.156559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnrm\" (UniqueName: \"kubernetes.io/projected/35af09b3-7c0b-4371-87eb-9c6a3c338d2b-kube-api-access-rpnrm\") pod \"cert-manager-operator-controller-manager-55c444bf86-zzkgn\" (UID: \"35af09b3-7c0b-4371-87eb-9c6a3c338d2b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.329837 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.329748 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" Sep 11 11:39:02.606183 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.606113 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn"] Sep 11 11:39:02.610015 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:39:02.609986 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35af09b3_7c0b_4371_87eb_9c6a3c338d2b.slice/crio-0445dd705b63cf69aee4d865e3710e3bd5d2a8b5502fb8914e9ebc0e4e055f5e WatchSource:0}: Error finding container 0445dd705b63cf69aee4d865e3710e3bd5d2a8b5502fb8914e9ebc0e4e055f5e: Status 404 returned error can't find the container with id 0445dd705b63cf69aee4d865e3710e3bd5d2a8b5502fb8914e9ebc0e4e055f5e Sep 11 11:39:02.612617 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:02.612601 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:39:03.383808 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:03.383767 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" event={"ID":"35af09b3-7c0b-4371-87eb-9c6a3c338d2b","Type":"ContainerStarted","Data":"0445dd705b63cf69aee4d865e3710e3bd5d2a8b5502fb8914e9ebc0e4e055f5e"} Sep 11 11:39:05.396096 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:05.396010 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" event={"ID":"35af09b3-7c0b-4371-87eb-9c6a3c338d2b","Type":"ContainerStarted","Data":"2b7901e686fd7ea63f68748b79a829ddc5dfae0bf0be6020891c60d4abfa715a"} Sep 11 11:39:05.422484 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:05.422436 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zzkgn" podStartSLOduration=2.258033949 podStartE2EDuration="4.422420048s" totalImagesPullingTime="2.164386099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:39:05.421122528 +0000 UTC m=+307.055715661" watchObservedRunningTime="2026-09-11 11:39:05.422420048 +0000 UTC m=+307.057013181" Sep 11 11:39:06.530519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.530490 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp"] Sep 11 11:39:06.532976 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.532960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.535601 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.535577 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4t6v\"" Sep 11 11:39:06.535601 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.535590 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 11:39:06.535798 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.535577 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 11:39:06.543153 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.543124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp"] Sep 11 11:39:06.576489 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.576451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhfv2\" (UniqueName: \"kubernetes.io/projected/05338178-202d-4c7c-8aa2-11f20c062878-kube-api-access-fhfv2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.576489 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.576494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.576701 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.576534 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.677528 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.677490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.677720 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.677538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.677720 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.677609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhfv2\" (UniqueName: \"kubernetes.io/projected/05338178-202d-4c7c-8aa2-11f20c062878-kube-api-access-fhfv2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.677908 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.677885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.677967 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.677909 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.694116 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.694083 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhfv2\" (UniqueName: \"kubernetes.io/projected/05338178-202d-4c7c-8aa2-11f20c062878-kube-api-access-fhfv2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:06.842207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:06.842125 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:07.113852 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:07.113765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp"] Sep 11 11:39:07.117575 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:39:07.117549 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05338178_202d_4c7c_8aa2_11f20c062878.slice/crio-bf271b049b560dcba40ea0ff39b25bb07bdec0c13aed723858d71617ca81b192 WatchSource:0}: Error finding container bf271b049b560dcba40ea0ff39b25bb07bdec0c13aed723858d71617ca81b192: Status 404 returned error can't find the container with id bf271b049b560dcba40ea0ff39b25bb07bdec0c13aed723858d71617ca81b192 Sep 11 11:39:07.407490 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:07.407456 2580 generic.go:356] "Generic (PLEG): container finished" podID="05338178-202d-4c7c-8aa2-11f20c062878" containerID="6433f83672c0d823086f73f28aae0ea1c48eac88b853739adcde6d1f71dab138" exitCode=0 Sep 11 11:39:07.407655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:07.407534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" event={"ID":"05338178-202d-4c7c-8aa2-11f20c062878","Type":"ContainerDied","Data":"6433f83672c0d823086f73f28aae0ea1c48eac88b853739adcde6d1f71dab138"} Sep 11 11:39:07.407655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:07.407572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" event={"ID":"05338178-202d-4c7c-8aa2-11f20c062878","Type":"ContainerStarted","Data":"bf271b049b560dcba40ea0ff39b25bb07bdec0c13aed723858d71617ca81b192"} Sep 11 11:39:08.292072 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.292024 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-62xff"] Sep 11 11:39:08.294389 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.294374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.297053 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.297018 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 11:39:08.297182 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.297126 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 11:39:08.297452 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.297435 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-n9vd2\"" Sep 11 11:39:08.303594 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.303573 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-62xff"] Sep 11 11:39:08.391186 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.391145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmcp4\" (UniqueName: \"kubernetes.io/projected/ffee6521-2b70-4438-82e1-752181edf8dc-kube-api-access-xmcp4\") pod \"cert-manager-cainjector-559cdd8c87-62xff\" (UID: \"ffee6521-2b70-4438-82e1-752181edf8dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.391186 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.391189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffee6521-2b70-4438-82e1-752181edf8dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-62xff\" (UID: \"ffee6521-2b70-4438-82e1-752181edf8dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.492274 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.492239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffee6521-2b70-4438-82e1-752181edf8dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-62xff\" (UID: \"ffee6521-2b70-4438-82e1-752181edf8dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.492439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.492319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmcp4\" (UniqueName: \"kubernetes.io/projected/ffee6521-2b70-4438-82e1-752181edf8dc-kube-api-access-xmcp4\") pod \"cert-manager-cainjector-559cdd8c87-62xff\" (UID: \"ffee6521-2b70-4438-82e1-752181edf8dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.500866 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.500822 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ffee6521-2b70-4438-82e1-752181edf8dc-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-62xff\" (UID: \"ffee6521-2b70-4438-82e1-752181edf8dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.501020 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.500910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmcp4\" (UniqueName: \"kubernetes.io/projected/ffee6521-2b70-4438-82e1-752181edf8dc-kube-api-access-xmcp4\") pod \"cert-manager-cainjector-559cdd8c87-62xff\" (UID: \"ffee6521-2b70-4438-82e1-752181edf8dc\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.625030 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.624939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" Sep 11 11:39:08.904913 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:08.904873 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-62xff"] Sep 11 11:39:08.908791 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:39:08.908755 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffee6521_2b70_4438_82e1_752181edf8dc.slice/crio-c377d984a4910387dbc298260d9fa0affeee24a5d6524029dc03319aa702eb84 WatchSource:0}: Error finding container c377d984a4910387dbc298260d9fa0affeee24a5d6524029dc03319aa702eb84: Status 404 returned error can't find the container with id c377d984a4910387dbc298260d9fa0affeee24a5d6524029dc03319aa702eb84 Sep 11 11:39:09.417381 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:09.417342 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" event={"ID":"ffee6521-2b70-4438-82e1-752181edf8dc","Type":"ContainerStarted","Data":"c377d984a4910387dbc298260d9fa0affeee24a5d6524029dc03319aa702eb84"} Sep 11 11:39:10.425095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:10.425057 2580 generic.go:356] "Generic (PLEG): container finished" podID="05338178-202d-4c7c-8aa2-11f20c062878" containerID="8a3c76f72c8ea8a3d43dec7e74c4bcc10a9885b6d1eaa7e8ad22126c6a8f9fe5" exitCode=0 Sep 11 11:39:10.425557 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:10.425115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" event={"ID":"05338178-202d-4c7c-8aa2-11f20c062878","Type":"ContainerDied","Data":"8a3c76f72c8ea8a3d43dec7e74c4bcc10a9885b6d1eaa7e8ad22126c6a8f9fe5"} Sep 11 11:39:11.434355 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:11.434315 2580 generic.go:356] "Generic (PLEG): container finished" podID="05338178-202d-4c7c-8aa2-11f20c062878" containerID="e38a0d4eb9787e45706c35635f98b0af1103efa54d7bd2a332eb80cb84cb50b1" exitCode=0 Sep 11 11:39:11.434795 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:11.434415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" event={"ID":"05338178-202d-4c7c-8aa2-11f20c062878","Type":"ContainerDied","Data":"e38a0d4eb9787e45706c35635f98b0af1103efa54d7bd2a332eb80cb84cb50b1"} Sep 11 11:39:12.592091 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.592053 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:12.627194 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.627109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-util\") pod \"05338178-202d-4c7c-8aa2-11f20c062878\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " Sep 11 11:39:12.627194 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.627152 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-bundle\") pod \"05338178-202d-4c7c-8aa2-11f20c062878\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " Sep 11 11:39:12.627400 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.627203 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhfv2\" (UniqueName: \"kubernetes.io/projected/05338178-202d-4c7c-8aa2-11f20c062878-kube-api-access-fhfv2\") pod \"05338178-202d-4c7c-8aa2-11f20c062878\" (UID: \"05338178-202d-4c7c-8aa2-11f20c062878\") " Sep 11 11:39:12.627638 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.627611 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-bundle" (OuterVolumeSpecName: "bundle") pod "05338178-202d-4c7c-8aa2-11f20c062878" (UID: "05338178-202d-4c7c-8aa2-11f20c062878"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:39:12.629499 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.629476 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05338178-202d-4c7c-8aa2-11f20c062878-kube-api-access-fhfv2" (OuterVolumeSpecName: "kube-api-access-fhfv2") pod "05338178-202d-4c7c-8aa2-11f20c062878" (UID: "05338178-202d-4c7c-8aa2-11f20c062878"). InnerVolumeSpecName "kube-api-access-fhfv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:39:12.728215 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.728183 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhfv2\" (UniqueName: \"kubernetes.io/projected/05338178-202d-4c7c-8aa2-11f20c062878-kube-api-access-fhfv2\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:12.728215 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.728211 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:12.820353 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.820321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-util" (OuterVolumeSpecName: "util") pod "05338178-202d-4c7c-8aa2-11f20c062878" (UID: "05338178-202d-4c7c-8aa2-11f20c062878"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:39:12.828968 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:12.828943 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05338178-202d-4c7c-8aa2-11f20c062878-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:13.445299 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:13.445269 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" Sep 11 11:39:13.445299 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:13.445276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4mhjp" event={"ID":"05338178-202d-4c7c-8aa2-11f20c062878","Type":"ContainerDied","Data":"bf271b049b560dcba40ea0ff39b25bb07bdec0c13aed723858d71617ca81b192"} Sep 11 11:39:13.445516 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:13.445324 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf271b049b560dcba40ea0ff39b25bb07bdec0c13aed723858d71617ca81b192" Sep 11 11:39:13.446795 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:13.446770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" event={"ID":"ffee6521-2b70-4438-82e1-752181edf8dc","Type":"ContainerStarted","Data":"24f21ac337d4f07c2c5972418f9da6af5997d15741942ffc0b00a831c10199a1"} Sep 11 11:39:13.465350 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:13.465304 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-62xff" podStartSLOduration=2.026131682 podStartE2EDuration="5.465288418s" totalImagesPullingTime="3.439156736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:39:13.463247231 +0000 UTC m=+315.097840363" watchObservedRunningTime="2026-09-11 11:39:13.465288418 +0000 UTC m=+315.099881551" Sep 11 11:39:16.995829 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.995794 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kwlnn"] Sep 11 11:39:16.996237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.996079 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05338178-202d-4c7c-8aa2-11f20c062878" containerName="util" Sep 11 11:39:16.996237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.996090 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05338178-202d-4c7c-8aa2-11f20c062878" containerName="util" Sep 11 11:39:16.996237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.996104 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05338178-202d-4c7c-8aa2-11f20c062878" containerName="pull" Sep 11 11:39:16.996237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.996109 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05338178-202d-4c7c-8aa2-11f20c062878" containerName="pull" Sep 11 11:39:16.996237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.996123 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05338178-202d-4c7c-8aa2-11f20c062878" containerName="extract" Sep 11 11:39:16.996237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.996128 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05338178-202d-4c7c-8aa2-11f20c062878" containerName="extract" Sep 11 11:39:16.996237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.996203 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05338178-202d-4c7c-8aa2-11f20c062878" containerName="extract" Sep 11 11:39:16.998354 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:16.998339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.000752 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.000729 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xmlrz\"" Sep 11 11:39:17.010895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.010872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kwlnn"] Sep 11 11:39:17.060937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.060904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4drr\" (UniqueName: \"kubernetes.io/projected/a36c91fc-e09b-4421-94d6-279149dbc458-kube-api-access-x4drr\") pod \"cert-manager-55b4d954d5-kwlnn\" (UID: \"a36c91fc-e09b-4421-94d6-279149dbc458\") " pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.061129 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.060949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a36c91fc-e09b-4421-94d6-279149dbc458-bound-sa-token\") pod \"cert-manager-55b4d954d5-kwlnn\" (UID: \"a36c91fc-e09b-4421-94d6-279149dbc458\") " pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.161976 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.161942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4drr\" (UniqueName: \"kubernetes.io/projected/a36c91fc-e09b-4421-94d6-279149dbc458-kube-api-access-x4drr\") pod \"cert-manager-55b4d954d5-kwlnn\" (UID: \"a36c91fc-e09b-4421-94d6-279149dbc458\") " pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.162137 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.161992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a36c91fc-e09b-4421-94d6-279149dbc458-bound-sa-token\") pod \"cert-manager-55b4d954d5-kwlnn\" (UID: \"a36c91fc-e09b-4421-94d6-279149dbc458\") " pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.171849 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.171817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a36c91fc-e09b-4421-94d6-279149dbc458-bound-sa-token\") pod \"cert-manager-55b4d954d5-kwlnn\" (UID: \"a36c91fc-e09b-4421-94d6-279149dbc458\") " pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.172199 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.172176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4drr\" (UniqueName: \"kubernetes.io/projected/a36c91fc-e09b-4421-94d6-279149dbc458-kube-api-access-x4drr\") pod \"cert-manager-55b4d954d5-kwlnn\" (UID: \"a36c91fc-e09b-4421-94d6-279149dbc458\") " pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.307403 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.307317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kwlnn" Sep 11 11:39:17.576505 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:17.576119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kwlnn"] Sep 11 11:39:17.579087 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:39:17.579057 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda36c91fc_e09b_4421_94d6_279149dbc458.slice/crio-1432ebe402e91ac0c108b25e45037f0c5c37924163d0836caeb819f9b18e20d3 WatchSource:0}: Error finding container 1432ebe402e91ac0c108b25e45037f0c5c37924163d0836caeb819f9b18e20d3: Status 404 returned error can't find the container with id 1432ebe402e91ac0c108b25e45037f0c5c37924163d0836caeb819f9b18e20d3 Sep 11 11:39:18.469376 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:18.469346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kwlnn" event={"ID":"a36c91fc-e09b-4421-94d6-279149dbc458","Type":"ContainerStarted","Data":"2465cb8da8adacd1f9c5326858279ff9167416ed726d33a43ea5bf9728d78305"} Sep 11 11:39:18.469727 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:18.469389 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kwlnn" event={"ID":"a36c91fc-e09b-4421-94d6-279149dbc458","Type":"ContainerStarted","Data":"1432ebe402e91ac0c108b25e45037f0c5c37924163d0836caeb819f9b18e20d3"} Sep 11 11:39:18.484357 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:18.484303 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kwlnn" podStartSLOduration=2.484285811 podStartE2EDuration="2.484285811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:39:18.483696645 +0000 UTC m=+320.118289777" watchObservedRunningTime="2026-09-11 11:39:18.484285811 +0000 UTC m=+320.118878944" Sep 11 11:39:27.308816 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.308784 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd"] Sep 11 11:39:27.311391 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.311374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.313782 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.313759 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4t6v\"" Sep 11 11:39:27.313782 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.313781 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 11:39:27.314722 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.314706 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 11:39:27.321671 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.321648 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd"] Sep 11 11:39:27.442055 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.442002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.442250 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.442072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.442250 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.442169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492gd\" (UniqueName: \"kubernetes.io/projected/ff86bf6b-d684-4c02-8b3b-3859677497fd-kube-api-access-492gd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.543318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.543285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.543469 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.543323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492gd\" (UniqueName: \"kubernetes.io/projected/ff86bf6b-d684-4c02-8b3b-3859677497fd-kube-api-access-492gd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.543469 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.543359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.543656 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.543636 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.543717 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.543699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.551629 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.551609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-492gd\" (UniqueName: \"kubernetes.io/projected/ff86bf6b-d684-4c02-8b3b-3859677497fd-kube-api-access-492gd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.621408 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.621341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:27.892461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:27.892436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd"] Sep 11 11:39:28.513688 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:28.513652 2580 generic.go:356] "Generic (PLEG): container finished" podID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerID="af48a762f35c04206fa1edc2e49c9086d6848b7ea379908abceb5b73f563ac73" exitCode=0 Sep 11 11:39:28.514077 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:28.513740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" event={"ID":"ff86bf6b-d684-4c02-8b3b-3859677497fd","Type":"ContainerDied","Data":"af48a762f35c04206fa1edc2e49c9086d6848b7ea379908abceb5b73f563ac73"} Sep 11 11:39:28.514077 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:28.513814 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" event={"ID":"ff86bf6b-d684-4c02-8b3b-3859677497fd","Type":"ContainerStarted","Data":"79b4a333372d53746ba5bd48cd3553094face517cb159917441e6e0087487862"} Sep 11 11:39:29.519655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:29.519575 2580 generic.go:356] "Generic (PLEG): container finished" podID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerID="21dd4bea44281a95f9bd9dcf069fd6b607492dc95621520846c4494a7ebc402c" exitCode=0 Sep 11 11:39:29.520021 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:29.519660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" event={"ID":"ff86bf6b-d684-4c02-8b3b-3859677497fd","Type":"ContainerDied","Data":"21dd4bea44281a95f9bd9dcf069fd6b607492dc95621520846c4494a7ebc402c"} Sep 11 11:39:30.526576 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:30.526543 2580 generic.go:356] "Generic (PLEG): container finished" podID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerID="64ec054de0fe49fdf096dbc54c2ad9c5ccc07c42db10bbb45976db050edeb2fe" exitCode=0 Sep 11 11:39:30.526576 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:30.526579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" event={"ID":"ff86bf6b-d684-4c02-8b3b-3859677497fd","Type":"ContainerDied","Data":"64ec054de0fe49fdf096dbc54c2ad9c5ccc07c42db10bbb45976db050edeb2fe"} Sep 11 11:39:31.651173 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.651153 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:31.775874 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.775844 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-492gd\" (UniqueName: \"kubernetes.io/projected/ff86bf6b-d684-4c02-8b3b-3859677497fd-kube-api-access-492gd\") pod \"ff86bf6b-d684-4c02-8b3b-3859677497fd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " Sep 11 11:39:31.776065 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.775906 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-bundle\") pod \"ff86bf6b-d684-4c02-8b3b-3859677497fd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " Sep 11 11:39:31.776065 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.775943 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-util\") pod \"ff86bf6b-d684-4c02-8b3b-3859677497fd\" (UID: \"ff86bf6b-d684-4c02-8b3b-3859677497fd\") " Sep 11 11:39:31.776684 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.776655 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-bundle" (OuterVolumeSpecName: "bundle") pod "ff86bf6b-d684-4c02-8b3b-3859677497fd" (UID: "ff86bf6b-d684-4c02-8b3b-3859677497fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:39:31.778002 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.777979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff86bf6b-d684-4c02-8b3b-3859677497fd-kube-api-access-492gd" (OuterVolumeSpecName: "kube-api-access-492gd") pod "ff86bf6b-d684-4c02-8b3b-3859677497fd" (UID: "ff86bf6b-d684-4c02-8b3b-3859677497fd"). InnerVolumeSpecName "kube-api-access-492gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:39:31.781804 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.781748 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-util" (OuterVolumeSpecName: "util") pod "ff86bf6b-d684-4c02-8b3b-3859677497fd" (UID: "ff86bf6b-d684-4c02-8b3b-3859677497fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:39:31.876906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.876867 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492gd\" (UniqueName: \"kubernetes.io/projected/ff86bf6b-d684-4c02-8b3b-3859677497fd-kube-api-access-492gd\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:31.876906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.876905 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:31.876906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:31.876914 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff86bf6b-d684-4c02-8b3b-3859677497fd-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:32.536483 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:32.536456 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" Sep 11 11:39:32.536677 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:32.536453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afb7bdd" event={"ID":"ff86bf6b-d684-4c02-8b3b-3859677497fd","Type":"ContainerDied","Data":"79b4a333372d53746ba5bd48cd3553094face517cb159917441e6e0087487862"} Sep 11 11:39:32.536677 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:32.536595 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b4a333372d53746ba5bd48cd3553094face517cb159917441e6e0087487862" Sep 11 11:39:38.379730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.379688 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj"] Sep 11 11:39:38.380215 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.380158 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerName="util" Sep 11 11:39:38.380215 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.380176 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerName="util" Sep 11 11:39:38.380215 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.380190 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerName="extract" Sep 11 11:39:38.380215 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.380197 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerName="extract" Sep 11 11:39:38.380415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.380229 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerName="pull" Sep 11 11:39:38.380415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.380239 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerName="pull" Sep 11 11:39:38.380415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.380326 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff86bf6b-d684-4c02-8b3b-3859677497fd" containerName="extract" Sep 11 11:39:38.385630 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.385608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.389674 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.389656 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 11:39:38.389894 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.389882 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 11:39:38.390403 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.390390 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4t6v\"" Sep 11 11:39:38.399083 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.399061 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj"] Sep 11 11:39:38.530051 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.530003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.530226 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.530062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.530226 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.530153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ng5k\" (UniqueName: \"kubernetes.io/projected/ddbe26ec-b740-45be-bf99-aa9fe800211e-kube-api-access-5ng5k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.631480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.631388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ng5k\" (UniqueName: \"kubernetes.io/projected/ddbe26ec-b740-45be-bf99-aa9fe800211e-kube-api-access-5ng5k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.631480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.631433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.631480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.631453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.631825 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.631811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.631866 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.631814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.645277 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.645249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ng5k\" (UniqueName: \"kubernetes.io/projected/ddbe26ec-b740-45be-bf99-aa9fe800211e-kube-api-access-5ng5k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.694549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.694518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:38.976264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:38.976230 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj"] Sep 11 11:39:38.979220 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:39:38.979189 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbe26ec_b740_45be_bf99_aa9fe800211e.slice/crio-10b17c2952313f37658f4f23b04cf3136c4490b380d67d36a1dffad485f1659e WatchSource:0}: Error finding container 10b17c2952313f37658f4f23b04cf3136c4490b380d67d36a1dffad485f1659e: Status 404 returned error can't find the container with id 10b17c2952313f37658f4f23b04cf3136c4490b380d67d36a1dffad485f1659e Sep 11 11:39:39.569048 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.569011 2580 generic.go:356] "Generic (PLEG): container finished" podID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerID="62dca7eab62fbc67512aa9b06e69aa669a30dc3b03efe150612e3cbfcad61c9a" exitCode=0 Sep 11 11:39:39.569405 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.569105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" event={"ID":"ddbe26ec-b740-45be-bf99-aa9fe800211e","Type":"ContainerDied","Data":"62dca7eab62fbc67512aa9b06e69aa669a30dc3b03efe150612e3cbfcad61c9a"} Sep 11 11:39:39.569405 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.569153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" event={"ID":"ddbe26ec-b740-45be-bf99-aa9fe800211e","Type":"ContainerStarted","Data":"10b17c2952313f37658f4f23b04cf3136c4490b380d67d36a1dffad485f1659e"} Sep 11 11:39:39.990909 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.990872 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p"] Sep 11 11:39:39.994683 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.994667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:39.997462 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.997442 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 11:39:39.997607 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.997542 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 11:39:39.997988 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.997970 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 11:39:39.998115 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.998056 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 11:39:39.998519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:39.998501 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jxm97\"" Sep 11 11:39:40.015193 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.015170 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p"] Sep 11 11:39:40.142969 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.142937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9227851a-51c5-4558-9d9f-a28c5a792e70-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.142969 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.142972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9227851a-51c5-4558-9d9f-a28c5a792e70-manifests\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.143192 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.142993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9227851a-51c5-4558-9d9f-a28c5a792e70-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.143192 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.143015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9227851a-51c5-4558-9d9f-a28c5a792e70-tmp\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.143192 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.143096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbd2\" (UniqueName: \"kubernetes.io/projected/9227851a-51c5-4558-9d9f-a28c5a792e70-kube-api-access-4vbd2\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.244196 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.244112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9227851a-51c5-4558-9d9f-a28c5a792e70-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.244196 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.244157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9227851a-51c5-4558-9d9f-a28c5a792e70-tmp\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.244196 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.244182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbd2\" (UniqueName: \"kubernetes.io/projected/9227851a-51c5-4558-9d9f-a28c5a792e70-kube-api-access-4vbd2\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.244451 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.244232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9227851a-51c5-4558-9d9f-a28c5a792e70-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.244451 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.244249 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9227851a-51c5-4558-9d9f-a28c5a792e70-manifests\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.244608 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.244590 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9227851a-51c5-4558-9d9f-a28c5a792e70-manifests\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.246561 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.246532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9227851a-51c5-4558-9d9f-a28c5a792e70-tmp\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.246751 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.246735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9227851a-51c5-4558-9d9f-a28c5a792e70-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.246799 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.246751 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9227851a-51c5-4558-9d9f-a28c5a792e70-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.268461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.268435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbd2\" (UniqueName: \"kubernetes.io/projected/9227851a-51c5-4558-9d9f-a28c5a792e70-kube-api-access-4vbd2\") pod \"opendatahub-operator-controller-manager-86f769d976-vnk2p\" (UID: \"9227851a-51c5-4558-9d9f-a28c5a792e70\") " pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.304455 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.304430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:40.583210 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:40.583132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p"] Sep 11 11:39:40.586736 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:39:40.586710 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9227851a_51c5_4558_9d9f_a28c5a792e70.slice/crio-39e4390fd980aaed25c814190e9460b9c0d902aa1656a06c3fc16f4b52a7dcbc WatchSource:0}: Error finding container 39e4390fd980aaed25c814190e9460b9c0d902aa1656a06c3fc16f4b52a7dcbc: Status 404 returned error can't find the container with id 39e4390fd980aaed25c814190e9460b9c0d902aa1656a06c3fc16f4b52a7dcbc Sep 11 11:39:41.583230 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:41.583202 2580 generic.go:356] "Generic (PLEG): container finished" podID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerID="326d9575da9545f415fb65941c565bc292203775bc9aad10cab4e30acd931760" exitCode=0 Sep 11 11:39:41.583616 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:41.583296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" event={"ID":"ddbe26ec-b740-45be-bf99-aa9fe800211e","Type":"ContainerDied","Data":"326d9575da9545f415fb65941c565bc292203775bc9aad10cab4e30acd931760"} Sep 11 11:39:41.585899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:41.585851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" event={"ID":"9227851a-51c5-4558-9d9f-a28c5a792e70","Type":"ContainerStarted","Data":"39e4390fd980aaed25c814190e9460b9c0d902aa1656a06c3fc16f4b52a7dcbc"} Sep 11 11:39:42.592435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:42.592403 2580 generic.go:356] "Generic (PLEG): container finished" podID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerID="b777c03e9086dac1961818638a8cfd393f5b4c4c6c224a7977d0d289286ff6f0" exitCode=0 Sep 11 11:39:42.592811 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:42.592475 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" event={"ID":"ddbe26ec-b740-45be-bf99-aa9fe800211e","Type":"ContainerDied","Data":"b777c03e9086dac1961818638a8cfd393f5b4c4c6c224a7977d0d289286ff6f0"} Sep 11 11:39:43.820029 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:43.819987 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:43.974838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:43.974808 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ng5k\" (UniqueName: \"kubernetes.io/projected/ddbe26ec-b740-45be-bf99-aa9fe800211e-kube-api-access-5ng5k\") pod \"ddbe26ec-b740-45be-bf99-aa9fe800211e\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " Sep 11 11:39:43.974990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:43.974861 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-util\") pod \"ddbe26ec-b740-45be-bf99-aa9fe800211e\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " Sep 11 11:39:43.974990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:43.974920 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-bundle\") pod \"ddbe26ec-b740-45be-bf99-aa9fe800211e\" (UID: \"ddbe26ec-b740-45be-bf99-aa9fe800211e\") " Sep 11 11:39:43.975637 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:43.975610 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-bundle" (OuterVolumeSpecName: "bundle") pod "ddbe26ec-b740-45be-bf99-aa9fe800211e" (UID: "ddbe26ec-b740-45be-bf99-aa9fe800211e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:39:43.976975 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:43.976948 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbe26ec-b740-45be-bf99-aa9fe800211e-kube-api-access-5ng5k" (OuterVolumeSpecName: "kube-api-access-5ng5k") pod "ddbe26ec-b740-45be-bf99-aa9fe800211e" (UID: "ddbe26ec-b740-45be-bf99-aa9fe800211e"). InnerVolumeSpecName "kube-api-access-5ng5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:39:43.980223 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:43.980204 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-util" (OuterVolumeSpecName: "util") pod "ddbe26ec-b740-45be-bf99-aa9fe800211e" (UID: "ddbe26ec-b740-45be-bf99-aa9fe800211e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:39:44.076234 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:44.076207 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:44.076234 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:44.076229 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ng5k\" (UniqueName: \"kubernetes.io/projected/ddbe26ec-b740-45be-bf99-aa9fe800211e-kube-api-access-5ng5k\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:44.076234 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:44.076238 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddbe26ec-b740-45be-bf99-aa9fe800211e-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:39:44.603427 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:44.603397 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" Sep 11 11:39:44.603427 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:44.603413 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rh6bj" event={"ID":"ddbe26ec-b740-45be-bf99-aa9fe800211e","Type":"ContainerDied","Data":"10b17c2952313f37658f4f23b04cf3136c4490b380d67d36a1dffad485f1659e"} Sep 11 11:39:44.603635 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:44.603452 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10b17c2952313f37658f4f23b04cf3136c4490b380d67d36a1dffad485f1659e" Sep 11 11:39:44.605283 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:44.605259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" event={"ID":"9227851a-51c5-4558-9d9f-a28c5a792e70","Type":"ContainerStarted","Data":"686a582431c3e284148761e5d039b944e4070f4f8837d8f68611aaa10836d0c8"} Sep 11 11:39:45.610941 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:45.610911 2580 generic.go:356] "Generic (PLEG): container finished" podID="9227851a-51c5-4558-9d9f-a28c5a792e70" containerID="686a582431c3e284148761e5d039b944e4070f4f8837d8f68611aaa10836d0c8" exitCode=0 Sep 11 11:39:45.611335 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:45.610951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" event={"ID":"9227851a-51c5-4558-9d9f-a28c5a792e70","Type":"ContainerDied","Data":"686a582431c3e284148761e5d039b944e4070f4f8837d8f68611aaa10836d0c8"} Sep 11 11:39:47.622122 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:47.622023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" event={"ID":"9227851a-51c5-4558-9d9f-a28c5a792e70","Type":"ContainerStarted","Data":"acd49ddae6eb5706ff7d10752cb7afaf9c14f172140d7d0e89a43f3eaedaed34"} Sep 11 11:39:47.622535 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:47.622176 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:47.646661 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:47.646612 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" podStartSLOduration=2.733904984 podStartE2EDuration="8.646598295s" totalImagesPullingTime="4.872067919s" totalInitContainerRuntime="1.040625392s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:39:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:39:47.644342042 +0000 UTC m=+349.278935174" watchObservedRunningTime="2026-09-11 11:39:47.646598295 +0000 UTC m=+349.281191489" Sep 11 11:39:57.484831 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.484759 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g"] Sep 11 11:39:57.485191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.485069 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerName="extract" Sep 11 11:39:57.485191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.485081 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerName="extract" Sep 11 11:39:57.485191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.485093 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerName="util" Sep 11 11:39:57.485191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.485099 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerName="util" Sep 11 11:39:57.485191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.485111 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerName="pull" Sep 11 11:39:57.485191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.485116 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerName="pull" Sep 11 11:39:57.485191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.485173 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbe26ec-b740-45be-bf99-aa9fe800211e" containerName="extract" Sep 11 11:39:57.488254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.488235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.491339 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.491311 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 11:39:57.491445 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.491344 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 11:39:57.491445 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.491377 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 11:39:57.491612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.491510 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 11:39:57.491673 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.491613 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zhdth\"" Sep 11 11:39:57.491726 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.491683 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 11:39:57.499290 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.499267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g"] Sep 11 11:39:57.675053 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.675011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0efba69d-c832-4a4b-9980-68816f34af46-manager-config\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.675203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.675066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0efba69d-c832-4a4b-9980-68816f34af46-cert\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.675203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.675151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0efba69d-c832-4a4b-9980-68816f34af46-metrics-cert\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.675203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.675190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fhtj\" (UniqueName: \"kubernetes.io/projected/0efba69d-c832-4a4b-9980-68816f34af46-kube-api-access-6fhtj\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.775654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.775575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fhtj\" (UniqueName: \"kubernetes.io/projected/0efba69d-c832-4a4b-9980-68816f34af46-kube-api-access-6fhtj\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.775654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.775617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0efba69d-c832-4a4b-9980-68816f34af46-manager-config\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.775844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.775668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0efba69d-c832-4a4b-9980-68816f34af46-cert\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.775844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.775727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0efba69d-c832-4a4b-9980-68816f34af46-metrics-cert\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.776429 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.776406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0efba69d-c832-4a4b-9980-68816f34af46-manager-config\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.778346 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.778321 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0efba69d-c832-4a4b-9980-68816f34af46-cert\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.778450 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.778361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0efba69d-c832-4a4b-9980-68816f34af46-metrics-cert\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.783527 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.783507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fhtj\" (UniqueName: \"kubernetes.io/projected/0efba69d-c832-4a4b-9980-68816f34af46-kube-api-access-6fhtj\") pod \"lws-controller-manager-77956949cf-8bk7g\" (UID: \"0efba69d-c832-4a4b-9980-68816f34af46\") " pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:57.797395 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:57.797373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:39:58.070354 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:58.070279 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g"] Sep 11 11:39:58.073488 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:39:58.073452 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efba69d_c832_4a4b_9980_68816f34af46.slice/crio-0748510b483cbe1be1a7bcdb09065a591fb63dce6baf232efd394a8cbd1b1286 WatchSource:0}: Error finding container 0748510b483cbe1be1a7bcdb09065a591fb63dce6baf232efd394a8cbd1b1286: Status 404 returned error can't find the container with id 0748510b483cbe1be1a7bcdb09065a591fb63dce6baf232efd394a8cbd1b1286 Sep 11 11:39:58.629101 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:58.629073 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86f769d976-vnk2p" Sep 11 11:39:58.674343 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:58.674310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" event={"ID":"0efba69d-c832-4a4b-9980-68816f34af46","Type":"ContainerStarted","Data":"0748510b483cbe1be1a7bcdb09065a591fb63dce6baf232efd394a8cbd1b1286"} Sep 11 11:39:59.680397 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:59.680356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" event={"ID":"0efba69d-c832-4a4b-9980-68816f34af46","Type":"ContainerStarted","Data":"b22614369331ca6479f0aa07d691268a338561dd61042328e67a003e72618b31"} Sep 11 11:39:59.680794 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:39:59.680438 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:40:10.687740 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:10.687704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" Sep 11 11:40:10.704478 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:10.704422 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77956949cf-8bk7g" podStartSLOduration=12.266995126 podStartE2EDuration="13.704406789s" totalImagesPullingTime="1.437411663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:39:59.722893688 +0000 UTC m=+361.357486821" watchObservedRunningTime="2026-09-11 11:40:10.704406789 +0000 UTC m=+372.339000000" Sep 11 11:40:11.153336 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.153303 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-584lz"] Sep 11 11:40:11.156858 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.156837 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.159423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.159401 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 11:40:11.159539 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.159463 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-k9mpr\"" Sep 11 11:40:11.170005 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.169984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-584lz"] Sep 11 11:40:11.175880 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.175857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7877e7b4-0bd3-4672-8271-7a74f775da30-cert\") pod \"odh-model-controller-7998c67dc4-584lz\" (UID: \"7877e7b4-0bd3-4672-8271-7a74f775da30\") " pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.175987 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.175893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/7877e7b4-0bd3-4672-8271-7a74f775da30-kube-api-access-xtmbf\") pod \"odh-model-controller-7998c67dc4-584lz\" (UID: \"7877e7b4-0bd3-4672-8271-7a74f775da30\") " pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.276500 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.276465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7877e7b4-0bd3-4672-8271-7a74f775da30-cert\") pod \"odh-model-controller-7998c67dc4-584lz\" (UID: \"7877e7b4-0bd3-4672-8271-7a74f775da30\") " pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.276642 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.276515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/7877e7b4-0bd3-4672-8271-7a74f775da30-kube-api-access-xtmbf\") pod \"odh-model-controller-7998c67dc4-584lz\" (UID: \"7877e7b4-0bd3-4672-8271-7a74f775da30\") " pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.276642 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:40:11.276617 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 11:40:11.276714 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:40:11.276680 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7877e7b4-0bd3-4672-8271-7a74f775da30-cert podName:7877e7b4-0bd3-4672-8271-7a74f775da30 nodeName:}" failed. No retries permitted until 2026-09-11 11:40:11.776661761 +0000 UTC m=+373.411254872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7877e7b4-0bd3-4672-8271-7a74f775da30-cert") pod "odh-model-controller-7998c67dc4-584lz" (UID: "7877e7b4-0bd3-4672-8271-7a74f775da30") : secret "odh-model-controller-webhook-cert" not found Sep 11 11:40:11.285652 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.285625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtmbf\" (UniqueName: \"kubernetes.io/projected/7877e7b4-0bd3-4672-8271-7a74f775da30-kube-api-access-xtmbf\") pod \"odh-model-controller-7998c67dc4-584lz\" (UID: \"7877e7b4-0bd3-4672-8271-7a74f775da30\") " pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.780517 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.780481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7877e7b4-0bd3-4672-8271-7a74f775da30-cert\") pod \"odh-model-controller-7998c67dc4-584lz\" (UID: \"7877e7b4-0bd3-4672-8271-7a74f775da30\") " pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.783122 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.783098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7877e7b4-0bd3-4672-8271-7a74f775da30-cert\") pod \"odh-model-controller-7998c67dc4-584lz\" (UID: \"7877e7b4-0bd3-4672-8271-7a74f775da30\") " pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:11.834341 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.834305 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4tsxp"] Sep 11 11:40:11.837673 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.837652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:11.840094 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.840075 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 11:40:11.856078 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.856055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4tsxp"] Sep 11 11:40:11.881614 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.881558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcp4\" (UniqueName: \"kubernetes.io/projected/6147e320-8c30-40a7-a4b1-a5b797d567a6-kube-api-access-zzcp4\") pod \"model-serving-api-54d89bd5fc-4tsxp\" (UID: \"6147e320-8c30-40a7-a4b1-a5b797d567a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:11.881855 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.881831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6147e320-8c30-40a7-a4b1-a5b797d567a6-tls-certs\") pod \"model-serving-api-54d89bd5fc-4tsxp\" (UID: \"6147e320-8c30-40a7-a4b1-a5b797d567a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:11.982450 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.982413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcp4\" (UniqueName: \"kubernetes.io/projected/6147e320-8c30-40a7-a4b1-a5b797d567a6-kube-api-access-zzcp4\") pod \"model-serving-api-54d89bd5fc-4tsxp\" (UID: \"6147e320-8c30-40a7-a4b1-a5b797d567a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:11.982605 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.982459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6147e320-8c30-40a7-a4b1-a5b797d567a6-tls-certs\") pod \"model-serving-api-54d89bd5fc-4tsxp\" (UID: \"6147e320-8c30-40a7-a4b1-a5b797d567a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:11.985040 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.985012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6147e320-8c30-40a7-a4b1-a5b797d567a6-tls-certs\") pod \"model-serving-api-54d89bd5fc-4tsxp\" (UID: \"6147e320-8c30-40a7-a4b1-a5b797d567a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:11.994644 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:11.994620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcp4\" (UniqueName: \"kubernetes.io/projected/6147e320-8c30-40a7-a4b1-a5b797d567a6-kube-api-access-zzcp4\") pod \"model-serving-api-54d89bd5fc-4tsxp\" (UID: \"6147e320-8c30-40a7-a4b1-a5b797d567a6\") " pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:12.067529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:12.067454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:12.148693 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:12.148667 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:12.361333 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:12.361309 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-584lz"] Sep 11 11:40:12.363085 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:40:12.363054 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7877e7b4_0bd3_4672_8271_7a74f775da30.slice/crio-eb1039a3693b90c2a5a6709c2e148a5aecd539bc302ff2726e08010bef0f3f4a WatchSource:0}: Error finding container eb1039a3693b90c2a5a6709c2e148a5aecd539bc302ff2726e08010bef0f3f4a: Status 404 returned error can't find the container with id eb1039a3693b90c2a5a6709c2e148a5aecd539bc302ff2726e08010bef0f3f4a Sep 11 11:40:12.426073 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:12.426022 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4tsxp"] Sep 11 11:40:12.427704 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:40:12.427676 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6147e320_8c30_40a7_a4b1_a5b797d567a6.slice/crio-9f0c42b31f09916dd0f6acde50226cfc3f1496e1d61baea49637d233a43d8548 WatchSource:0}: Error finding container 9f0c42b31f09916dd0f6acde50226cfc3f1496e1d61baea49637d233a43d8548: Status 404 returned error can't find the container with id 9f0c42b31f09916dd0f6acde50226cfc3f1496e1d61baea49637d233a43d8548 Sep 11 11:40:12.742923 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:12.742889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-584lz" event={"ID":"7877e7b4-0bd3-4672-8271-7a74f775da30","Type":"ContainerStarted","Data":"eb1039a3693b90c2a5a6709c2e148a5aecd539bc302ff2726e08010bef0f3f4a"} Sep 11 11:40:12.743965 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:12.743942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" event={"ID":"6147e320-8c30-40a7-a4b1-a5b797d567a6","Type":"ContainerStarted","Data":"9f0c42b31f09916dd0f6acde50226cfc3f1496e1d61baea49637d233a43d8548"} Sep 11 11:40:16.492207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.492168 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw"] Sep 11 11:40:16.495316 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.495293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.497963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.497937 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 11:40:16.498173 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.498005 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9wdwc\"" Sep 11 11:40:16.507351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.507327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw"] Sep 11 11:40:16.523614 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523632 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsm6\" (UniqueName: \"kubernetes.io/projected/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-kube-api-access-hwsm6\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.523902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.523893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624390 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsm6\" (UniqueName: \"kubernetes.io/projected/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-kube-api-access-hwsm6\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624789 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624764 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624861 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624861 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.624958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.624933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.625522 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.625456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.625522 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.625454 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.625805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.625780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.625885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.625851 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.627095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.627075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.627176 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.627165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.632304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.632282 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.632461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.632440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsm6\" (UniqueName: \"kubernetes.io/projected/5487c2f7-ffbd-4e85-a8eb-6feae50e4821-kube-api-access-hwsm6\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw\" (UID: \"5487c2f7-ffbd-4e85-a8eb-6feae50e4821\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:16.769820 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.769717 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" event={"ID":"6147e320-8c30-40a7-a4b1-a5b797d567a6","Type":"ContainerStarted","Data":"bf23c09e2d0922910d6b977c390aa0f0d90d989ad68f4c58b269f900bdb66d68"} Sep 11 11:40:16.769820 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.769792 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:16.771295 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.771271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-584lz" event={"ID":"7877e7b4-0bd3-4672-8271-7a74f775da30","Type":"ContainerStarted","Data":"fc42e684ad0c1725070ddbd6087504aea1cec427914d151223be63d994a2cd04"} Sep 11 11:40:16.771436 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.771394 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:16.785678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.785639 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" podStartSLOduration=1.645948788 podStartE2EDuration="5.78562449s" totalImagesPullingTime="4.139675702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:40:16.784275448 +0000 UTC m=+378.418868595" watchObservedRunningTime="2026-09-11 11:40:16.78562449 +0000 UTC m=+378.420217650" Sep 11 11:40:16.802506 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.802463 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-584lz" podStartSLOduration=1.5715321869999999 podStartE2EDuration="5.802447636s" totalImagesPullingTime="4.230915449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:40:16.800229341 +0000 UTC m=+378.434822473" watchObservedRunningTime="2026-09-11 11:40:16.802447636 +0000 UTC m=+378.437040819" Sep 11 11:40:16.810054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:16.810015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:17.094861 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:17.094836 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw"] Sep 11 11:40:17.096908 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:40:17.096877 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5487c2f7_ffbd_4e85_a8eb_6feae50e4821.slice/crio-aa402ffc388ebb70b319a8d08d754a4ba8aad934757ce252040bed53472a17aa WatchSource:0}: Error finding container aa402ffc388ebb70b319a8d08d754a4ba8aad934757ce252040bed53472a17aa: Status 404 returned error can't find the container with id aa402ffc388ebb70b319a8d08d754a4ba8aad934757ce252040bed53472a17aa Sep 11 11:40:17.777699 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:17.777657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" event={"ID":"5487c2f7-ffbd-4e85-a8eb-6feae50e4821","Type":"ContainerStarted","Data":"aa402ffc388ebb70b319a8d08d754a4ba8aad934757ce252040bed53472a17aa"} Sep 11 11:40:21.241560 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.241507 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:40:21.241882 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.241615 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:40:21.241882 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.241660 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:40:21.459821 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.459785 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj"] Sep 11 11:40:21.463749 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.463731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.467447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.467415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rxxw5\"" Sep 11 11:40:21.468313 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.468155 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 11:40:21.494937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.494873 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj"] Sep 11 11:40:21.540261 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.540230 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh"] Sep 11 11:40:21.544494 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.544475 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:21.547743 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.547723 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d5mpf\"" Sep 11 11:40:21.553435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.553415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 11:40:21.562524 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.562501 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh"] Sep 11 11:40:21.569846 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.569823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mmxq\" (UniqueName: \"kubernetes.io/projected/95b350c5-520e-4ea8-955f-d7fa2d83ae7f-kube-api-access-4mmxq\") pod \"kserve-controller-manager-d6f4b65b8-nd2wj\" (UID: \"95b350c5-520e-4ea8-955f-d7fa2d83ae7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.569949 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.569853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38de991-95e5-4937-8e7c-9f758d3508bf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-54qdh\" (UID: \"b38de991-95e5-4937-8e7c-9f758d3508bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:21.569949 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.569873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95b350c5-520e-4ea8-955f-d7fa2d83ae7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-nd2wj\" (UID: \"95b350c5-520e-4ea8-955f-d7fa2d83ae7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.570051 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.569939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swv9h\" (UniqueName: \"kubernetes.io/projected/b38de991-95e5-4937-8e7c-9f758d3508bf-kube-api-access-swv9h\") pod \"llmisvc-controller-manager-777c6bd7d5-54qdh\" (UID: \"b38de991-95e5-4937-8e7c-9f758d3508bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:21.671118 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.671087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mmxq\" (UniqueName: \"kubernetes.io/projected/95b350c5-520e-4ea8-955f-d7fa2d83ae7f-kube-api-access-4mmxq\") pod \"kserve-controller-manager-d6f4b65b8-nd2wj\" (UID: \"95b350c5-520e-4ea8-955f-d7fa2d83ae7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.671118 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.671123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38de991-95e5-4937-8e7c-9f758d3508bf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-54qdh\" (UID: \"b38de991-95e5-4937-8e7c-9f758d3508bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:21.671314 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.671153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95b350c5-520e-4ea8-955f-d7fa2d83ae7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-nd2wj\" (UID: \"95b350c5-520e-4ea8-955f-d7fa2d83ae7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.671314 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.671197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swv9h\" (UniqueName: \"kubernetes.io/projected/b38de991-95e5-4937-8e7c-9f758d3508bf-kube-api-access-swv9h\") pod \"llmisvc-controller-manager-777c6bd7d5-54qdh\" (UID: \"b38de991-95e5-4937-8e7c-9f758d3508bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:21.671314 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:40:21.671241 2580 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 11 11:40:21.671314 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:40:21.671299 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38de991-95e5-4937-8e7c-9f758d3508bf-cert podName:b38de991-95e5-4937-8e7c-9f758d3508bf nodeName:}" failed. No retries permitted until 2026-09-11 11:40:22.171285153 +0000 UTC m=+383.805878264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b38de991-95e5-4937-8e7c-9f758d3508bf-cert") pod "llmisvc-controller-manager-777c6bd7d5-54qdh" (UID: "b38de991-95e5-4937-8e7c-9f758d3508bf") : secret "llmisvc-webhook-server-cert" not found Sep 11 11:40:21.673849 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.673825 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95b350c5-520e-4ea8-955f-d7fa2d83ae7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-nd2wj\" (UID: \"95b350c5-520e-4ea8-955f-d7fa2d83ae7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.680978 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.680950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mmxq\" (UniqueName: \"kubernetes.io/projected/95b350c5-520e-4ea8-955f-d7fa2d83ae7f-kube-api-access-4mmxq\") pod \"kserve-controller-manager-d6f4b65b8-nd2wj\" (UID: \"95b350c5-520e-4ea8-955f-d7fa2d83ae7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.682207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.682191 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swv9h\" (UniqueName: \"kubernetes.io/projected/b38de991-95e5-4937-8e7c-9f758d3508bf-kube-api-access-swv9h\") pod \"llmisvc-controller-manager-777c6bd7d5-54qdh\" (UID: \"b38de991-95e5-4937-8e7c-9f758d3508bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:21.778173 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.778086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:21.801209 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.801172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" event={"ID":"5487c2f7-ffbd-4e85-a8eb-6feae50e4821","Type":"ContainerStarted","Data":"dad19750724c81162b28f62735e339af0a9db27b36fd68e0f083625a3544e235"} Sep 11 11:40:21.810992 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.810969 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:21.835342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:21.835288 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" podStartSLOduration=1.692991232 podStartE2EDuration="5.835261113s" totalImagesPullingTime="4.142269881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:40:21.83371968 +0000 UTC m=+383.468312817" watchObservedRunningTime="2026-09-11 11:40:21.835261113 +0000 UTC m=+383.469854247" Sep 11 11:40:22.081276 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.081198 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj"] Sep 11 11:40:22.084292 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:40:22.084266 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b350c5_520e_4ea8_955f_d7fa2d83ae7f.slice/crio-051ef01dd202207fd0f21af12606f36808ae8e6711c35e3ac1a1e9b3a51a61ae WatchSource:0}: Error finding container 051ef01dd202207fd0f21af12606f36808ae8e6711c35e3ac1a1e9b3a51a61ae: Status 404 returned error can't find the container with id 051ef01dd202207fd0f21af12606f36808ae8e6711c35e3ac1a1e9b3a51a61ae Sep 11 11:40:22.175095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.175063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38de991-95e5-4937-8e7c-9f758d3508bf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-54qdh\" (UID: \"b38de991-95e5-4937-8e7c-9f758d3508bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:22.177609 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.177589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38de991-95e5-4937-8e7c-9f758d3508bf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-54qdh\" (UID: \"b38de991-95e5-4937-8e7c-9f758d3508bf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:22.455462 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.455423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:22.727912 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.727888 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh"] Sep 11 11:40:22.729492 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:40:22.729465 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb38de991_95e5_4937_8e7c_9f758d3508bf.slice/crio-1808684cb189bf88d4d4e57c8043e92d4faafa0d20144a06c04dd434c1b80c59 WatchSource:0}: Error finding container 1808684cb189bf88d4d4e57c8043e92d4faafa0d20144a06c04dd434c1b80c59: Status 404 returned error can't find the container with id 1808684cb189bf88d4d4e57c8043e92d4faafa0d20144a06c04dd434c1b80c59 Sep 11 11:40:22.807225 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.807188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" event={"ID":"95b350c5-520e-4ea8-955f-d7fa2d83ae7f","Type":"ContainerStarted","Data":"051ef01dd202207fd0f21af12606f36808ae8e6711c35e3ac1a1e9b3a51a61ae"} Sep 11 11:40:22.808432 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.808405 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" event={"ID":"b38de991-95e5-4937-8e7c-9f758d3508bf","Type":"ContainerStarted","Data":"1808684cb189bf88d4d4e57c8043e92d4faafa0d20144a06c04dd434c1b80c59"} Sep 11 11:40:22.815457 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:22.815434 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:23.815056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:23.815003 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:23.816294 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:23.816271 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw" Sep 11 11:40:25.826986 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:25.826942 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" event={"ID":"95b350c5-520e-4ea8-955f-d7fa2d83ae7f","Type":"ContainerStarted","Data":"eb4c15e8717eebbc293f618b15665808808d0ec2edabbe53294384b0e0728b9f"} Sep 11 11:40:25.827439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:25.827007 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:25.828519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:25.828489 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" event={"ID":"b38de991-95e5-4937-8e7c-9f758d3508bf","Type":"ContainerStarted","Data":"09713b3a86d5c7e34d64a3fe8e27879da00405d3afe766fc4f7cd516943dd29b"} Sep 11 11:40:25.842404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:25.842323 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" podStartSLOduration=2.004812204 podStartE2EDuration="4.842308833s" totalImagesPullingTime="2.837496629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:40:25.842115135 +0000 UTC m=+387.476708268" watchObservedRunningTime="2026-09-11 11:40:25.842308833 +0000 UTC m=+387.476901967" Sep 11 11:40:25.859103 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:25.859059 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" podStartSLOduration=1.958395353 podStartE2EDuration="4.859023969s" totalImagesPullingTime="2.900628616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:40:25.857679707 +0000 UTC m=+387.492272853" watchObservedRunningTime="2026-09-11 11:40:25.859023969 +0000 UTC m=+387.493617101" Sep 11 11:40:26.833410 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:26.833382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:40:27.780734 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:27.780707 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-584lz" Sep 11 11:40:27.782480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:27.782454 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4tsxp" Sep 11 11:40:56.839112 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:56.839082 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nd2wj" Sep 11 11:40:57.840284 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:40:57.840257 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-54qdh" Sep 11 11:41:17.504398 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.504365 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk"] Sep 11 11:41:17.511995 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.511972 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.514368 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.514339 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 11:41:17.514503 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.514398 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4t6v\"" Sep 11 11:41:17.514869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.514853 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 11:41:17.516308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.516280 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk"] Sep 11 11:41:17.536714 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.536690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.536859 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.536721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c4xb\" (UniqueName: \"kubernetes.io/projected/27de70fa-2e40-455f-870a-396a68abed6a-kube-api-access-2c4xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.536859 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.536775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.599810 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.599780 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd"] Sep 11 11:41:17.602586 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.602570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.614663 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.614636 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd"] Sep 11 11:41:17.638108 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.638108 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c4xb\" (UniqueName: \"kubernetes.io/projected/27de70fa-2e40-455f-870a-396a68abed6a-kube-api-access-2c4xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.638300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.638300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.638300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.638300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5hc\" (UniqueName: \"kubernetes.io/projected/3b1dc058-e184-4236-bd1c-657b643482b1-kube-api-access-fd5hc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.638454 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638434 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.638518 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.638503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.647278 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.647252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c4xb\" (UniqueName: \"kubernetes.io/projected/27de70fa-2e40-455f-870a-396a68abed6a-kube-api-access-2c4xb\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.699897 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.699865 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj"] Sep 11 11:41:17.702494 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.702478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.718170 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.718138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj"] Sep 11 11:41:17.739292 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.739423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.739423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.739423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5mhc\" (UniqueName: \"kubernetes.io/projected/063a4841-46f5-4cf2-baad-b46488a15264-kube-api-access-q5mhc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.739556 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.739556 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5hc\" (UniqueName: \"kubernetes.io/projected/3b1dc058-e184-4236-bd1c-657b643482b1-kube-api-access-fd5hc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.739628 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.739770 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.739747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.748419 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.748398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5hc\" (UniqueName: \"kubernetes.io/projected/3b1dc058-e184-4236-bd1c-657b643482b1-kube-api-access-fd5hc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.815931 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.815862 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz"] Sep 11 11:41:17.818454 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.818439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.823840 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.823817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:17.830494 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.830475 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz"] Sep 11 11:41:17.840881 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.840857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.840989 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.840892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.840989 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.840917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5mhc\" (UniqueName: \"kubernetes.io/projected/063a4841-46f5-4cf2-baad-b46488a15264-kube-api-access-q5mhc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.840989 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.840953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.840989 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.840983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrht\" (UniqueName: \"kubernetes.io/projected/e3a921d3-5090-4f06-ba65-467b0aff3cea-kube-api-access-5mrht\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.841246 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.841061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.841310 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.841287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.841367 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.841311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.854398 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.854368 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5mhc\" (UniqueName: \"kubernetes.io/projected/063a4841-46f5-4cf2-baad-b46488a15264-kube-api-access-q5mhc\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:17.911425 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.911391 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:17.942277 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.941927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.942277 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.941974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrht\" (UniqueName: \"kubernetes.io/projected/e3a921d3-5090-4f06-ba65-467b0aff3cea-kube-api-access-5mrht\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.942277 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.942022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.942277 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.942222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.942490 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.942374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:17.956663 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:17.956623 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrht\" (UniqueName: \"kubernetes.io/projected/e3a921d3-5090-4f06-ba65-467b0aff3cea-kube-api-access-5mrht\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:18.012139 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:18.012110 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:18.100733 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:18.100710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk"] Sep 11 11:41:18.102769 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:41:18.102749 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27de70fa_2e40_455f_870a_396a68abed6a.slice/crio-fe8b82e612303244b685bce32a4bfa280f1eb71c1d3d74483582f298486e8af5 WatchSource:0}: Error finding container fe8b82e612303244b685bce32a4bfa280f1eb71c1d3d74483582f298486e8af5: Status 404 returned error can't find the container with id fe8b82e612303244b685bce32a4bfa280f1eb71c1d3d74483582f298486e8af5 Sep 11 11:41:18.128059 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:18.128021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:18.208435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:18.208410 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd"] Sep 11 11:41:18.210144 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:41:18.210114 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1dc058_e184_4236_bd1c_657b643482b1.slice/crio-03dc74cb3dda83eeb22d68873d4ae21c48f7dbdc3a14e6d44feaf55a3f52258f WatchSource:0}: Error finding container 03dc74cb3dda83eeb22d68873d4ae21c48f7dbdc3a14e6d44feaf55a3f52258f: Status 404 returned error can't find the container with id 03dc74cb3dda83eeb22d68873d4ae21c48f7dbdc3a14e6d44feaf55a3f52258f Sep 11 11:41:18.303470 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:18.303446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj"] Sep 11 11:41:18.307263 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:41:18.307243 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063a4841_46f5_4cf2_baad_b46488a15264.slice/crio-418c130a6403093849a4d8787694bc3242262ad98f7238d25435840b0c092616 WatchSource:0}: Error finding container 418c130a6403093849a4d8787694bc3242262ad98f7238d25435840b0c092616: Status 404 returned error can't find the container with id 418c130a6403093849a4d8787694bc3242262ad98f7238d25435840b0c092616 Sep 11 11:41:18.409894 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:18.409869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz"] Sep 11 11:41:18.414025 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:41:18.413995 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a921d3_5090_4f06_ba65_467b0aff3cea.slice/crio-8e88a5e35a7baf9c2c1c57dea1000f1df049a904307f26a7e36e3e07661ee2ef WatchSource:0}: Error finding container 8e88a5e35a7baf9c2c1c57dea1000f1df049a904307f26a7e36e3e07661ee2ef: Status 404 returned error can't find the container with id 8e88a5e35a7baf9c2c1c57dea1000f1df049a904307f26a7e36e3e07661ee2ef Sep 11 11:41:19.082346 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.082311 2580 generic.go:356] "Generic (PLEG): container finished" podID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerID="73e389dc401ffdc94d8039e57dc2faa02da032570e14ebb339101142119df291" exitCode=0 Sep 11 11:41:19.082772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.082392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" event={"ID":"e3a921d3-5090-4f06-ba65-467b0aff3cea","Type":"ContainerDied","Data":"73e389dc401ffdc94d8039e57dc2faa02da032570e14ebb339101142119df291"} Sep 11 11:41:19.082772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.082450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" event={"ID":"e3a921d3-5090-4f06-ba65-467b0aff3cea","Type":"ContainerStarted","Data":"8e88a5e35a7baf9c2c1c57dea1000f1df049a904307f26a7e36e3e07661ee2ef"} Sep 11 11:41:19.084172 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.084145 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b1dc058-e184-4236-bd1c-657b643482b1" containerID="9560e913f51fd48abb51c7503a14447ec628ff059c60bf6f67297824d922ec71" exitCode=0 Sep 11 11:41:19.084302 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.084173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" event={"ID":"3b1dc058-e184-4236-bd1c-657b643482b1","Type":"ContainerDied","Data":"9560e913f51fd48abb51c7503a14447ec628ff059c60bf6f67297824d922ec71"} Sep 11 11:41:19.084302 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.084222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" event={"ID":"3b1dc058-e184-4236-bd1c-657b643482b1","Type":"ContainerStarted","Data":"03dc74cb3dda83eeb22d68873d4ae21c48f7dbdc3a14e6d44feaf55a3f52258f"} Sep 11 11:41:19.086069 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.086025 2580 generic.go:356] "Generic (PLEG): container finished" podID="063a4841-46f5-4cf2-baad-b46488a15264" containerID="99147b0b1f4833be5993bd680861133ed8ce9b71f045a8c59423d66f059accdf" exitCode=0 Sep 11 11:41:19.086152 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.086072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" event={"ID":"063a4841-46f5-4cf2-baad-b46488a15264","Type":"ContainerDied","Data":"99147b0b1f4833be5993bd680861133ed8ce9b71f045a8c59423d66f059accdf"} Sep 11 11:41:19.086152 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.086131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" event={"ID":"063a4841-46f5-4cf2-baad-b46488a15264","Type":"ContainerStarted","Data":"418c130a6403093849a4d8787694bc3242262ad98f7238d25435840b0c092616"} Sep 11 11:41:19.088083 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.088026 2580 generic.go:356] "Generic (PLEG): container finished" podID="27de70fa-2e40-455f-870a-396a68abed6a" containerID="3ba7724b5a119226df9ff3225ecc93360dcb8f8b2fc9b337275c4679e864266f" exitCode=0 Sep 11 11:41:19.088184 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.088066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" event={"ID":"27de70fa-2e40-455f-870a-396a68abed6a","Type":"ContainerDied","Data":"3ba7724b5a119226df9ff3225ecc93360dcb8f8b2fc9b337275c4679e864266f"} Sep 11 11:41:19.088184 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:19.088153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" event={"ID":"27de70fa-2e40-455f-870a-396a68abed6a","Type":"ContainerStarted","Data":"fe8b82e612303244b685bce32a4bfa280f1eb71c1d3d74483582f298486e8af5"} Sep 11 11:41:20.095560 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:20.095476 2580 generic.go:356] "Generic (PLEG): container finished" podID="063a4841-46f5-4cf2-baad-b46488a15264" containerID="f286c0322f862705646eb5b1b0546e428facb7d8e2bcb60b31024689e43614c7" exitCode=0 Sep 11 11:41:20.095968 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:20.095563 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" event={"ID":"063a4841-46f5-4cf2-baad-b46488a15264","Type":"ContainerDied","Data":"f286c0322f862705646eb5b1b0546e428facb7d8e2bcb60b31024689e43614c7"} Sep 11 11:41:20.097598 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:20.097575 2580 generic.go:356] "Generic (PLEG): container finished" podID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerID="80bb48ec5b83f625e7338c3bac507167d877ae3a92e5edb01433b224ddc499e1" exitCode=0 Sep 11 11:41:20.097689 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:20.097649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" event={"ID":"e3a921d3-5090-4f06-ba65-467b0aff3cea","Type":"ContainerDied","Data":"80bb48ec5b83f625e7338c3bac507167d877ae3a92e5edb01433b224ddc499e1"} Sep 11 11:41:21.105381 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:21.105350 2580 generic.go:356] "Generic (PLEG): container finished" podID="063a4841-46f5-4cf2-baad-b46488a15264" containerID="f57d315d1bfc32bda9beee1874c1b57de3f9225cf4faae03e2f805e47f839f0a" exitCode=0 Sep 11 11:41:21.105709 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:21.105435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" event={"ID":"063a4841-46f5-4cf2-baad-b46488a15264","Type":"ContainerDied","Data":"f57d315d1bfc32bda9beee1874c1b57de3f9225cf4faae03e2f805e47f839f0a"} Sep 11 11:41:21.107583 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:21.107561 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" event={"ID":"27de70fa-2e40-455f-870a-396a68abed6a","Type":"ContainerStarted","Data":"bb90a2f920c3162f91d0a01fcedfdffbb4be244567a67d21c2768feab5374779"} Sep 11 11:41:21.110018 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:21.109993 2580 generic.go:356] "Generic (PLEG): container finished" podID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerID="e9f0c6cdf5c031f9fcc73348000f4b0361a1c9d1cbd199168353812f576c3683" exitCode=0 Sep 11 11:41:21.110115 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:21.110069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" event={"ID":"e3a921d3-5090-4f06-ba65-467b0aff3cea","Type":"ContainerDied","Data":"e9f0c6cdf5c031f9fcc73348000f4b0361a1c9d1cbd199168353812f576c3683"} Sep 11 11:41:22.116771 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.116735 2580 generic.go:356] "Generic (PLEG): container finished" podID="27de70fa-2e40-455f-870a-396a68abed6a" containerID="bb90a2f920c3162f91d0a01fcedfdffbb4be244567a67d21c2768feab5374779" exitCode=0 Sep 11 11:41:22.116771 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.116760 2580 generic.go:356] "Generic (PLEG): container finished" podID="27de70fa-2e40-455f-870a-396a68abed6a" containerID="cf7cd83a60e2d5843fa56baa127595cccd0400dd209608d7396576d80b3ebeac" exitCode=0 Sep 11 11:41:22.117230 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.116815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" event={"ID":"27de70fa-2e40-455f-870a-396a68abed6a","Type":"ContainerDied","Data":"bb90a2f920c3162f91d0a01fcedfdffbb4be244567a67d21c2768feab5374779"} Sep 11 11:41:22.117230 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.116862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" event={"ID":"27de70fa-2e40-455f-870a-396a68abed6a","Type":"ContainerDied","Data":"cf7cd83a60e2d5843fa56baa127595cccd0400dd209608d7396576d80b3ebeac"} Sep 11 11:41:22.270916 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.270889 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:22.273768 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.273750 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:22.384663 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.384590 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5mhc\" (UniqueName: \"kubernetes.io/projected/063a4841-46f5-4cf2-baad-b46488a15264-kube-api-access-q5mhc\") pod \"063a4841-46f5-4cf2-baad-b46488a15264\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " Sep 11 11:41:22.384663 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.384617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mrht\" (UniqueName: \"kubernetes.io/projected/e3a921d3-5090-4f06-ba65-467b0aff3cea-kube-api-access-5mrht\") pod \"e3a921d3-5090-4f06-ba65-467b0aff3cea\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " Sep 11 11:41:22.384663 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.384641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-bundle\") pod \"e3a921d3-5090-4f06-ba65-467b0aff3cea\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " Sep 11 11:41:22.384955 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.384686 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-util\") pod \"063a4841-46f5-4cf2-baad-b46488a15264\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " Sep 11 11:41:22.384955 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.384822 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-bundle\") pod \"063a4841-46f5-4cf2-baad-b46488a15264\" (UID: \"063a4841-46f5-4cf2-baad-b46488a15264\") " Sep 11 11:41:22.384955 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.384887 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-util\") pod \"e3a921d3-5090-4f06-ba65-467b0aff3cea\" (UID: \"e3a921d3-5090-4f06-ba65-467b0aff3cea\") " Sep 11 11:41:22.385488 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.385467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-bundle" (OuterVolumeSpecName: "bundle") pod "e3a921d3-5090-4f06-ba65-467b0aff3cea" (UID: "e3a921d3-5090-4f06-ba65-467b0aff3cea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:22.385565 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.385462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-bundle" (OuterVolumeSpecName: "bundle") pod "063a4841-46f5-4cf2-baad-b46488a15264" (UID: "063a4841-46f5-4cf2-baad-b46488a15264"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:22.387339 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.387314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a921d3-5090-4f06-ba65-467b0aff3cea-kube-api-access-5mrht" (OuterVolumeSpecName: "kube-api-access-5mrht") pod "e3a921d3-5090-4f06-ba65-467b0aff3cea" (UID: "e3a921d3-5090-4f06-ba65-467b0aff3cea"). InnerVolumeSpecName "kube-api-access-5mrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:41:22.387416 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.387356 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063a4841-46f5-4cf2-baad-b46488a15264-kube-api-access-q5mhc" (OuterVolumeSpecName: "kube-api-access-q5mhc") pod "063a4841-46f5-4cf2-baad-b46488a15264" (UID: "063a4841-46f5-4cf2-baad-b46488a15264"). InnerVolumeSpecName "kube-api-access-q5mhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:41:22.390679 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.390640 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-util" (OuterVolumeSpecName: "util") pod "063a4841-46f5-4cf2-baad-b46488a15264" (UID: "063a4841-46f5-4cf2-baad-b46488a15264"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:22.391258 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.391237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-util" (OuterVolumeSpecName: "util") pod "e3a921d3-5090-4f06-ba65-467b0aff3cea" (UID: "e3a921d3-5090-4f06-ba65-467b0aff3cea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:22.486027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.486000 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:22.486027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.486026 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/063a4841-46f5-4cf2-baad-b46488a15264-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:22.486027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.486056 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:22.486242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.486065 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5mhc\" (UniqueName: \"kubernetes.io/projected/063a4841-46f5-4cf2-baad-b46488a15264-kube-api-access-q5mhc\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:22.486242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.486075 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrht\" (UniqueName: \"kubernetes.io/projected/e3a921d3-5090-4f06-ba65-467b0aff3cea-kube-api-access-5mrht\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:22.486242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:22.486083 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3a921d3-5090-4f06-ba65-467b0aff3cea-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:23.123935 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.123909 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" Sep 11 11:41:23.123935 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.123927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c664ttj" event={"ID":"063a4841-46f5-4cf2-baad-b46488a15264","Type":"ContainerDied","Data":"418c130a6403093849a4d8787694bc3242262ad98f7238d25435840b0c092616"} Sep 11 11:41:23.124520 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.123976 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="418c130a6403093849a4d8787694bc3242262ad98f7238d25435840b0c092616" Sep 11 11:41:23.126159 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.126140 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" Sep 11 11:41:23.126296 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.126180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ac8lsz" event={"ID":"e3a921d3-5090-4f06-ba65-467b0aff3cea","Type":"ContainerDied","Data":"8e88a5e35a7baf9c2c1c57dea1000f1df049a904307f26a7e36e3e07661ee2ef"} Sep 11 11:41:23.126296 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.126219 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e88a5e35a7baf9c2c1c57dea1000f1df049a904307f26a7e36e3e07661ee2ef" Sep 11 11:41:23.257466 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.257443 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:23.392878 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.392840 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-bundle\") pod \"27de70fa-2e40-455f-870a-396a68abed6a\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " Sep 11 11:41:23.392878 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.392887 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-util\") pod \"27de70fa-2e40-455f-870a-396a68abed6a\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " Sep 11 11:41:23.393127 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.392930 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c4xb\" (UniqueName: \"kubernetes.io/projected/27de70fa-2e40-455f-870a-396a68abed6a-kube-api-access-2c4xb\") pod \"27de70fa-2e40-455f-870a-396a68abed6a\" (UID: \"27de70fa-2e40-455f-870a-396a68abed6a\") " Sep 11 11:41:23.393445 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.393423 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-bundle" (OuterVolumeSpecName: "bundle") pod "27de70fa-2e40-455f-870a-396a68abed6a" (UID: "27de70fa-2e40-455f-870a-396a68abed6a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:23.395260 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.395238 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27de70fa-2e40-455f-870a-396a68abed6a-kube-api-access-2c4xb" (OuterVolumeSpecName: "kube-api-access-2c4xb") pod "27de70fa-2e40-455f-870a-396a68abed6a" (UID: "27de70fa-2e40-455f-870a-396a68abed6a"). InnerVolumeSpecName "kube-api-access-2c4xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:41:23.398788 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.398756 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-util" (OuterVolumeSpecName: "util") pod "27de70fa-2e40-455f-870a-396a68abed6a" (UID: "27de70fa-2e40-455f-870a-396a68abed6a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:23.494003 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.493935 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:23.494003 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.493957 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27de70fa-2e40-455f-870a-396a68abed6a-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:23.494003 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:23.493967 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c4xb\" (UniqueName: \"kubernetes.io/projected/27de70fa-2e40-455f-870a-396a68abed6a-kube-api-access-2c4xb\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:24.133340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:24.133307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" event={"ID":"27de70fa-2e40-455f-870a-396a68abed6a","Type":"ContainerDied","Data":"fe8b82e612303244b685bce32a4bfa280f1eb71c1d3d74483582f298486e8af5"} Sep 11 11:41:24.133340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:24.133334 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7g6jbk" Sep 11 11:41:24.133340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:24.133351 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe8b82e612303244b685bce32a4bfa280f1eb71c1d3d74483582f298486e8af5" Sep 11 11:41:28.161206 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:28.161173 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b1dc058-e184-4236-bd1c-657b643482b1" containerID="452827c21afe7bbc2d46e9aa1cb1d9e9882c6e33d8d4232c960fb3a669e52c47" exitCode=0 Sep 11 11:41:28.161551 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:28.161238 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" event={"ID":"3b1dc058-e184-4236-bd1c-657b643482b1","Type":"ContainerDied","Data":"452827c21afe7bbc2d46e9aa1cb1d9e9882c6e33d8d4232c960fb3a669e52c47"} Sep 11 11:41:29.168669 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:29.168635 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b1dc058-e184-4236-bd1c-657b643482b1" containerID="cc2c62d271676f3fcfd6c8230e72aa566f032f06d42b1bcef03a891f6e40ce65" exitCode=0 Sep 11 11:41:29.169099 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:29.168715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" event={"ID":"3b1dc058-e184-4236-bd1c-657b643482b1","Type":"ContainerDied","Data":"cc2c62d271676f3fcfd6c8230e72aa566f032f06d42b1bcef03a891f6e40ce65"} Sep 11 11:41:30.309138 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.309110 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:30.448640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.448552 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd5hc\" (UniqueName: \"kubernetes.io/projected/3b1dc058-e184-4236-bd1c-657b643482b1-kube-api-access-fd5hc\") pod \"3b1dc058-e184-4236-bd1c-657b643482b1\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " Sep 11 11:41:30.448640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.448604 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-util\") pod \"3b1dc058-e184-4236-bd1c-657b643482b1\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " Sep 11 11:41:30.448867 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.448660 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-bundle\") pod \"3b1dc058-e184-4236-bd1c-657b643482b1\" (UID: \"3b1dc058-e184-4236-bd1c-657b643482b1\") " Sep 11 11:41:30.449283 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.449252 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-bundle" (OuterVolumeSpecName: "bundle") pod "3b1dc058-e184-4236-bd1c-657b643482b1" (UID: "3b1dc058-e184-4236-bd1c-657b643482b1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:30.450838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.450818 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1dc058-e184-4236-bd1c-657b643482b1-kube-api-access-fd5hc" (OuterVolumeSpecName: "kube-api-access-fd5hc") pod "3b1dc058-e184-4236-bd1c-657b643482b1" (UID: "3b1dc058-e184-4236-bd1c-657b643482b1"). InnerVolumeSpecName "kube-api-access-fd5hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:41:30.452869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.452844 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-util" (OuterVolumeSpecName: "util") pod "3b1dc058-e184-4236-bd1c-657b643482b1" (UID: "3b1dc058-e184-4236-bd1c-657b643482b1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:41:30.549844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.549795 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-bundle\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:30.549844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.549837 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5hc\" (UniqueName: \"kubernetes.io/projected/3b1dc058-e184-4236-bd1c-657b643482b1-kube-api-access-fd5hc\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:30.549844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:30.549849 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b1dc058-e184-4236-bd1c-657b643482b1-util\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:41:31.180742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:31.180715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" event={"ID":"3b1dc058-e184-4236-bd1c-657b643482b1","Type":"ContainerDied","Data":"03dc74cb3dda83eeb22d68873d4ae21c48f7dbdc3a14e6d44feaf55a3f52258f"} Sep 11 11:41:31.180906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:31.180765 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03dc74cb3dda83eeb22d68873d4ae21c48f7dbdc3a14e6d44feaf55a3f52258f" Sep 11 11:41:31.180906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:31.180720 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmzcxd" Sep 11 11:41:55.909531 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909486 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws"] Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909848 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27de70fa-2e40-455f-870a-396a68abed6a" containerName="util" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909862 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27de70fa-2e40-455f-870a-396a68abed6a" containerName="util" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909869 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063a4841-46f5-4cf2-baad-b46488a15264" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909874 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="063a4841-46f5-4cf2-baad-b46488a15264" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909886 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063a4841-46f5-4cf2-baad-b46488a15264" containerName="util" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909891 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="063a4841-46f5-4cf2-baad-b46488a15264" containerName="util" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909898 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27de70fa-2e40-455f-870a-396a68abed6a" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909903 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27de70fa-2e40-455f-870a-396a68abed6a" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909911 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27de70fa-2e40-455f-870a-396a68abed6a" containerName="extract" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909916 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="27de70fa-2e40-455f-870a-396a68abed6a" containerName="extract" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909928 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909936 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909943 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063a4841-46f5-4cf2-baad-b46488a15264" containerName="extract" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909947 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="063a4841-46f5-4cf2-baad-b46488a15264" containerName="extract" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909955 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1dc058-e184-4236-bd1c-657b643482b1" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909959 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1dc058-e184-4236-bd1c-657b643482b1" containerName="pull" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909971 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1dc058-e184-4236-bd1c-657b643482b1" containerName="util" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909976 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1dc058-e184-4236-bd1c-657b643482b1" containerName="util" Sep 11 11:41:55.909974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909982 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerName="util" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909988 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerName="util" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909993 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerName="extract" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.909999 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerName="extract" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.910009 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1dc058-e184-4236-bd1c-657b643482b1" containerName="extract" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.910017 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1dc058-e184-4236-bd1c-657b643482b1" containerName="extract" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.910095 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="063a4841-46f5-4cf2-baad-b46488a15264" containerName="extract" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.910104 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a921d3-5090-4f06-ba65-467b0aff3cea" containerName="extract" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.910110 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="27de70fa-2e40-455f-870a-396a68abed6a" containerName="extract" Sep 11 11:41:55.910570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.910117 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1dc058-e184-4236-bd1c-657b643482b1" containerName="extract" Sep 11 11:41:55.914393 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.914373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:55.918255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.918228 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 11 11:41:55.918396 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.918379 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 11 11:41:55.918472 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.918463 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 11:41:55.919173 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.919159 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 11:41:55.919267 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.919250 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4nnwd\"" Sep 11 11:41:55.930834 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:55.930810 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws"] Sep 11 11:41:56.074450 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.074415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzs77\" (UniqueName: \"kubernetes.io/projected/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-kube-api-access-rzs77\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.074612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.074465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.074612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.074525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.175284 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.175193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzs77\" (UniqueName: \"kubernetes.io/projected/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-kube-api-access-rzs77\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.175284 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.175248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.175500 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.175295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.175868 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.175845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.177857 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.177833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.188694 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.188666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzs77\" (UniqueName: \"kubernetes.io/projected/6d223b6f-0d34-4b65-a2a3-e9a914399a3e-kube-api-access-rzs77\") pod \"kuadrant-console-plugin-fbdd4f445-8jhws\" (UID: \"6d223b6f-0d34-4b65-a2a3-e9a914399a3e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.223347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.223312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" Sep 11 11:41:56.497785 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:56.497716 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws"] Sep 11 11:41:56.500788 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:41:56.500763 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d223b6f_0d34_4b65_a2a3_e9a914399a3e.slice/crio-540dc33e2924d57d15ecf799c1215744f334b5197a9287d6edf6e8da5212c51d WatchSource:0}: Error finding container 540dc33e2924d57d15ecf799c1215744f334b5197a9287d6edf6e8da5212c51d: Status 404 returned error can't find the container with id 540dc33e2924d57d15ecf799c1215744f334b5197a9287d6edf6e8da5212c51d Sep 11 11:41:57.316687 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:41:57.316650 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" event={"ID":"6d223b6f-0d34-4b65-a2a3-e9a914399a3e","Type":"ContainerStarted","Data":"540dc33e2924d57d15ecf799c1215744f334b5197a9287d6edf6e8da5212c51d"} Sep 11 11:42:01.352436 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:01.352399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" event={"ID":"6d223b6f-0d34-4b65-a2a3-e9a914399a3e","Type":"ContainerStarted","Data":"3273afa94d953c9a3e7a7b6741f32ce6fca0f8542e671c6813436a218bd009e6"} Sep 11 11:42:01.369808 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:01.369766 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8jhws" podStartSLOduration=1.7205809090000002 podStartE2EDuration="6.369751413s" totalImagesPullingTime="4.649170504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:42:01.367708805 +0000 UTC m=+483.002301938" watchObservedRunningTime="2026-09-11 11:42:01.369751413 +0000 UTC m=+483.004344545" Sep 11 11:42:06.290481 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.290446 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf"] Sep 11 11:42:06.294851 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.294833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.301355 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.301243 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8dpsl\"" Sep 11 11:42:06.315724 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.315703 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf"] Sep 11 11:42:06.357017 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.356992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvx5h\" (UniqueName: \"kubernetes.io/projected/cb705e24-9489-49c6-ab8a-5a0b09f55e89-kube-api-access-wvx5h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.357165 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.357028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb705e24-9489-49c6-ab8a-5a0b09f55e89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.457988 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.457953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvx5h\" (UniqueName: \"kubernetes.io/projected/cb705e24-9489-49c6-ab8a-5a0b09f55e89-kube-api-access-wvx5h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.457988 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.457993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb705e24-9489-49c6-ab8a-5a0b09f55e89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.458347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.458331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb705e24-9489-49c6-ab8a-5a0b09f55e89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.469902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.469878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvx5h\" (UniqueName: \"kubernetes.io/projected/cb705e24-9489-49c6-ab8a-5a0b09f55e89-kube-api-access-wvx5h\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.593365 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.593288 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf"] Sep 11 11:42:06.593592 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.593577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:06.601465 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.601439 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf"] Sep 11 11:42:06.631450 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.631416 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5"] Sep 11 11:42:06.637939 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.637921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:06.664168 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.664127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q28d\" (UniqueName: \"kubernetes.io/projected/2ea2260b-30d3-4126-8338-23ce8224abc8-kube-api-access-5q28d\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-st7r5\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:06.664329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.664210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea2260b-30d3-4126-8338-23ce8224abc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-st7r5\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:06.666817 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.666785 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5"] Sep 11 11:42:06.765699 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.765660 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q28d\" (UniqueName: \"kubernetes.io/projected/2ea2260b-30d3-4126-8338-23ce8224abc8-kube-api-access-5q28d\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-st7r5\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:06.765876 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.765717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea2260b-30d3-4126-8338-23ce8224abc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-st7r5\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:06.766180 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.766163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea2260b-30d3-4126-8338-23ce8224abc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-st7r5\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:06.785619 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.785591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q28d\" (UniqueName: \"kubernetes.io/projected/2ea2260b-30d3-4126-8338-23ce8224abc8-kube-api-access-5q28d\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-st7r5\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:06.948804 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:06.948768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:07.235185 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:07.235162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5"] Sep 11 11:42:07.237020 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:42:07.236989 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea2260b_30d3_4126_8338_23ce8224abc8.slice/crio-7f768c4f405413c9275030afbf5cb77c96a0161fe0eca4b70289123cd3b81433 WatchSource:0}: Error finding container 7f768c4f405413c9275030afbf5cb77c96a0161fe0eca4b70289123cd3b81433: Status 404 returned error can't find the container with id 7f768c4f405413c9275030afbf5cb77c96a0161fe0eca4b70289123cd3b81433 Sep 11 11:42:07.383126 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:07.383092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" event={"ID":"2ea2260b-30d3-4126-8338-23ce8224abc8","Type":"ContainerStarted","Data":"7f768c4f405413c9275030afbf5cb77c96a0161fe0eca4b70289123cd3b81433"} Sep 11 11:42:09.526424 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:42:09.526375 2580 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-lcmqf_kuadrant-system_cb705e24-9489-49c6-ab8a-5a0b09f55e89_0(e55f11f078959e7376a7978fa1d71be1ae725dd8072db967ad4f72088702fb55): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-lcmqf to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Sep 11 11:42:09.526814 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:42:09.526456 2580 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-lcmqf_kuadrant-system_cb705e24-9489-49c6-ab8a-5a0b09f55e89_0(e55f11f078959e7376a7978fa1d71be1ae725dd8072db967ad4f72088702fb55): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-lcmqf to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:10.405720 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.405691 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:10.408384 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:42:10.408343 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="cb705e24-9489-49c6-ab8a-5a0b09f55e89" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:10.413452 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.413430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:10.416848 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:42:10.416822 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="cb705e24-9489-49c6-ab8a-5a0b09f55e89" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:10.494213 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.494178 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvx5h\" (UniqueName: \"kubernetes.io/projected/cb705e24-9489-49c6-ab8a-5a0b09f55e89-kube-api-access-wvx5h\") pod \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " Sep 11 11:42:10.494388 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.494242 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb705e24-9489-49c6-ab8a-5a0b09f55e89-extensions-socket-volume\") pod \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\" (UID: \"cb705e24-9489-49c6-ab8a-5a0b09f55e89\") " Sep 11 11:42:10.494589 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.494564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb705e24-9489-49c6-ab8a-5a0b09f55e89-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cb705e24-9489-49c6-ab8a-5a0b09f55e89" (UID: "cb705e24-9489-49c6-ab8a-5a0b09f55e89"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:42:10.496835 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.496808 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb705e24-9489-49c6-ab8a-5a0b09f55e89-kube-api-access-wvx5h" (OuterVolumeSpecName: "kube-api-access-wvx5h") pod "cb705e24-9489-49c6-ab8a-5a0b09f55e89" (UID: "cb705e24-9489-49c6-ab8a-5a0b09f55e89"). InnerVolumeSpecName "kube-api-access-wvx5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:42:10.594878 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.594841 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb705e24-9489-49c6-ab8a-5a0b09f55e89-extensions-socket-volume\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:42:10.594878 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.594873 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvx5h\" (UniqueName: \"kubernetes.io/projected/cb705e24-9489-49c6-ab8a-5a0b09f55e89-kube-api-access-wvx5h\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:42:10.874838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:10.874753 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb705e24-9489-49c6-ab8a-5a0b09f55e89" path="/var/lib/kubelet/pods/cb705e24-9489-49c6-ab8a-5a0b09f55e89/volumes" Sep 11 11:42:11.412723 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:11.412674 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" event={"ID":"2ea2260b-30d3-4126-8338-23ce8224abc8","Type":"ContainerStarted","Data":"8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de"} Sep 11 11:42:11.412723 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:11.412713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:11.412963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:11.412772 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:11.438341 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:11.438294 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" podStartSLOduration=1.573668909 podStartE2EDuration="5.438281696s" totalImagesPullingTime="3.864612787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:42:11.436626824 +0000 UTC m=+493.071219956" watchObservedRunningTime="2026-09-11 11:42:11.438281696 +0000 UTC m=+493.072874830" Sep 11 11:42:11.438477 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:42:11.438387 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="cb705e24-9489-49c6-ab8a-5a0b09f55e89" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:11.440087 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:42:11.440064 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lcmqf\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="cb705e24-9489-49c6-ab8a-5a0b09f55e89" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lcmqf" Sep 11 11:42:22.420000 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:22.419969 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:38.976608 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:38.976581 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8"] Sep 11 11:42:38.995797 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:38.995773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8"] Sep 11 11:42:38.995952 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:38.995899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:38.998210 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:38.998188 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v5lvf\"" Sep 11 11:42:39.133873 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.133832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.133879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.133903 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0233a32a-dd49-4369-974e-3e82d7103a23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.133926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0233a32a-dd49-4369-974e-3e82d7103a23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.133954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtspw\" (UniqueName: \"kubernetes.io/projected/0233a32a-dd49-4369-974e-3e82d7103a23-kube-api-access-gtspw\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.134009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0233a32a-dd49-4369-974e-3e82d7103a23-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134301 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.134085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134301 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.134102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.134301 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.134169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235372 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtspw\" (UniqueName: \"kubernetes.io/projected/0233a32a-dd49-4369-974e-3e82d7103a23-kube-api-access-gtspw\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235372 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0233a32a-dd49-4369-974e-3e82d7103a23-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0233a32a-dd49-4369-974e-3e82d7103a23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235655 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0233a32a-dd49-4369-974e-3e82d7103a23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.235984 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.235919 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.236199 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.236176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.236291 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.236214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.236291 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.236258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.236519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.236498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0233a32a-dd49-4369-974e-3e82d7103a23-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.238123 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.238102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0233a32a-dd49-4369-974e-3e82d7103a23-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.238677 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.238661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0233a32a-dd49-4369-974e-3e82d7103a23-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.244825 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.244799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtspw\" (UniqueName: \"kubernetes.io/projected/0233a32a-dd49-4369-974e-3e82d7103a23-kube-api-access-gtspw\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.245162 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.245143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0233a32a-dd49-4369-974e-3e82d7103a23-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-l8td8\" (UID: \"0233a32a-dd49-4369-974e-3e82d7103a23\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.306307 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.306282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:39.592187 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.590125 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8"] Sep 11 11:42:39.597413 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.597378 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:42:39.597528 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.597453 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:42:39.597528 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:39.597494 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:42:40.564893 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:40.564860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" event={"ID":"0233a32a-dd49-4369-974e-3e82d7103a23","Type":"ContainerStarted","Data":"d052f732901e4268b8383e411545cf492e0c12f0e3346eb9b140aabd0b2b5668"} Sep 11 11:42:40.565304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:40.564913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" event={"ID":"0233a32a-dd49-4369-974e-3e82d7103a23","Type":"ContainerStarted","Data":"82b4ba8beb9c7205d0668a94fd6fea355843e16cc322cf1f75db597e5e238594"} Sep 11 11:42:40.586266 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:40.586225 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" podStartSLOduration=2.586209629 podStartE2EDuration="2.586209629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:42:40.585429447 +0000 UTC m=+522.220022570" watchObservedRunningTime="2026-09-11 11:42:40.586209629 +0000 UTC m=+522.220802741" Sep 11 11:42:40.842309 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:40.842229 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5"] Sep 11 11:42:40.842465 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:40.842414 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" podUID="2ea2260b-30d3-4126-8338-23ce8224abc8" containerName="manager" containerID="cri-o://8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de" gracePeriod=10 Sep 11 11:42:41.095789 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.095729 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:41.152077 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.152051 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea2260b-30d3-4126-8338-23ce8224abc8-extensions-socket-volume\") pod \"2ea2260b-30d3-4126-8338-23ce8224abc8\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " Sep 11 11:42:41.152234 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.152090 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q28d\" (UniqueName: \"kubernetes.io/projected/2ea2260b-30d3-4126-8338-23ce8224abc8-kube-api-access-5q28d\") pod \"2ea2260b-30d3-4126-8338-23ce8224abc8\" (UID: \"2ea2260b-30d3-4126-8338-23ce8224abc8\") " Sep 11 11:42:41.152521 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.152499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea2260b-30d3-4126-8338-23ce8224abc8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2ea2260b-30d3-4126-8338-23ce8224abc8" (UID: "2ea2260b-30d3-4126-8338-23ce8224abc8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:42:41.154395 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.154373 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea2260b-30d3-4126-8338-23ce8224abc8-kube-api-access-5q28d" (OuterVolumeSpecName: "kube-api-access-5q28d") pod "2ea2260b-30d3-4126-8338-23ce8224abc8" (UID: "2ea2260b-30d3-4126-8338-23ce8224abc8"). InnerVolumeSpecName "kube-api-access-5q28d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:42:41.253433 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.253403 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q28d\" (UniqueName: \"kubernetes.io/projected/2ea2260b-30d3-4126-8338-23ce8224abc8-kube-api-access-5q28d\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:42:41.253433 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.253430 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ea2260b-30d3-4126-8338-23ce8224abc8-extensions-socket-volume\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:42:41.307356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.307319 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:41.312096 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.312071 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:41.571106 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.571073 2580 generic.go:356] "Generic (PLEG): container finished" podID="2ea2260b-30d3-4126-8338-23ce8224abc8" containerID="8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de" exitCode=0 Sep 11 11:42:41.571532 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.571148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" event={"ID":"2ea2260b-30d3-4126-8338-23ce8224abc8","Type":"ContainerDied","Data":"8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de"} Sep 11 11:42:41.571532 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.571165 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" Sep 11 11:42:41.571532 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.571204 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5" event={"ID":"2ea2260b-30d3-4126-8338-23ce8224abc8","Type":"ContainerDied","Data":"7f768c4f405413c9275030afbf5cb77c96a0161fe0eca4b70289123cd3b81433"} Sep 11 11:42:41.571532 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.571233 2580 scope.go:117] "RemoveContainer" containerID="8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de" Sep 11 11:42:41.571747 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.571683 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:41.572821 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.572802 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:42:41.583283 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.583265 2580 scope.go:117] "RemoveContainer" containerID="8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de" Sep 11 11:42:41.583875 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:42:41.583851 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de\": container with ID starting with 8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de not found: ID does not exist" containerID="8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de" Sep 11 11:42:41.583970 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.583885 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de"} err="failed to get container status \"8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de\": rpc error: code = NotFound desc = could not find container \"8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de\": container with ID starting with 8965a4bd3dd22d62e02ec6863c6fcafcec71888025e5e6e84b58178e8ad6f4de not found: ID does not exist" Sep 11 11:42:41.644846 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.644811 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5"] Sep 11 11:42:41.646891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:41.646869 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-st7r5"] Sep 11 11:42:42.872107 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:42:42.872075 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ea2260b-30d3-4126-8338-23ce8224abc8" path="/var/lib/kubelet/pods/2ea2260b-30d3-4126-8338-23ce8224abc8/volumes" Sep 11 11:43:00.541280 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.541240 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:43:00.541648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.541601 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea2260b-30d3-4126-8338-23ce8224abc8" containerName="manager" Sep 11 11:43:00.541648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.541611 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea2260b-30d3-4126-8338-23ce8224abc8" containerName="manager" Sep 11 11:43:00.541717 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.541684 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ea2260b-30d3-4126-8338-23ce8224abc8" containerName="manager" Sep 11 11:43:00.544510 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.544492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:00.557157 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.557140 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 11:43:00.564979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.564951 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:43:00.623280 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.623253 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:43:00.723827 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.723796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87vf9\" (UniqueName: \"kubernetes.io/projected/5034794a-d3e7-4a1f-b7a5-8f23f9817239-kube-api-access-87vf9\") pod \"limitador-limitador-84994f7ffc-8cltk\" (UID: \"5034794a-d3e7-4a1f-b7a5-8f23f9817239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:00.724004 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.723847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5034794a-d3e7-4a1f-b7a5-8f23f9817239-config-file\") pod \"limitador-limitador-84994f7ffc-8cltk\" (UID: \"5034794a-d3e7-4a1f-b7a5-8f23f9817239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:00.824436 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.824352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5034794a-d3e7-4a1f-b7a5-8f23f9817239-config-file\") pod \"limitador-limitador-84994f7ffc-8cltk\" (UID: \"5034794a-d3e7-4a1f-b7a5-8f23f9817239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:00.824581 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.824435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87vf9\" (UniqueName: \"kubernetes.io/projected/5034794a-d3e7-4a1f-b7a5-8f23f9817239-kube-api-access-87vf9\") pod \"limitador-limitador-84994f7ffc-8cltk\" (UID: \"5034794a-d3e7-4a1f-b7a5-8f23f9817239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:00.824991 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.824973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5034794a-d3e7-4a1f-b7a5-8f23f9817239-config-file\") pod \"limitador-limitador-84994f7ffc-8cltk\" (UID: \"5034794a-d3e7-4a1f-b7a5-8f23f9817239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:00.839166 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.839143 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87vf9\" (UniqueName: \"kubernetes.io/projected/5034794a-d3e7-4a1f-b7a5-8f23f9817239-kube-api-access-87vf9\") pod \"limitador-limitador-84994f7ffc-8cltk\" (UID: \"5034794a-d3e7-4a1f-b7a5-8f23f9817239\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:00.855112 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:00.855090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:01.154361 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.154335 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:43:01.155201 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:43:01.155177 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5034794a_d3e7_4a1f_b7a5_8f23f9817239.slice/crio-c00d308095db300d73619b8526423dbea875e478e2fe1aa2b14bcd487f07da79 WatchSource:0}: Error finding container c00d308095db300d73619b8526423dbea875e478e2fe1aa2b14bcd487f07da79: Status 404 returned error can't find the container with id c00d308095db300d73619b8526423dbea875e478e2fe1aa2b14bcd487f07da79 Sep 11 11:43:01.300321 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.300287 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-8x8wg"] Sep 11 11:43:01.303570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.303545 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8x8wg" Sep 11 11:43:01.311542 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.311525 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qkq5f\"" Sep 11 11:43:01.322262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.322239 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8x8wg"] Sep 11 11:43:01.429606 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.429502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9tr\" (UniqueName: \"kubernetes.io/projected/60654a9b-1719-4c21-aae8-a91c831bd3b0-kube-api-access-rz9tr\") pod \"authorino-59c974bd48-8x8wg\" (UID: \"60654a9b-1719-4c21-aae8-a91c831bd3b0\") " pod="kuadrant-system/authorino-59c974bd48-8x8wg" Sep 11 11:43:01.530889 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.530857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9tr\" (UniqueName: \"kubernetes.io/projected/60654a9b-1719-4c21-aae8-a91c831bd3b0-kube-api-access-rz9tr\") pod \"authorino-59c974bd48-8x8wg\" (UID: \"60654a9b-1719-4c21-aae8-a91c831bd3b0\") " pod="kuadrant-system/authorino-59c974bd48-8x8wg" Sep 11 11:43:01.542567 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.542538 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9tr\" (UniqueName: \"kubernetes.io/projected/60654a9b-1719-4c21-aae8-a91c831bd3b0-kube-api-access-rz9tr\") pod \"authorino-59c974bd48-8x8wg\" (UID: \"60654a9b-1719-4c21-aae8-a91c831bd3b0\") " pod="kuadrant-system/authorino-59c974bd48-8x8wg" Sep 11 11:43:01.613624 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.613535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8x8wg" Sep 11 11:43:01.692823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.692248 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" event={"ID":"5034794a-d3e7-4a1f-b7a5-8f23f9817239","Type":"ContainerStarted","Data":"c00d308095db300d73619b8526423dbea875e478e2fe1aa2b14bcd487f07da79"} Sep 11 11:43:01.906232 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:01.906204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8x8wg"] Sep 11 11:43:01.942418 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:43:01.942386 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60654a9b_1719_4c21_aae8_a91c831bd3b0.slice/crio-1023e6d7f67b2ddd3173087a191ebc8a8613d5e386e41101bc5b93eb05cefc5f WatchSource:0}: Error finding container 1023e6d7f67b2ddd3173087a191ebc8a8613d5e386e41101bc5b93eb05cefc5f: Status 404 returned error can't find the container with id 1023e6d7f67b2ddd3173087a191ebc8a8613d5e386e41101bc5b93eb05cefc5f Sep 11 11:43:02.699300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:02.699264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8x8wg" event={"ID":"60654a9b-1719-4c21-aae8-a91c831bd3b0","Type":"ContainerStarted","Data":"1023e6d7f67b2ddd3173087a191ebc8a8613d5e386e41101bc5b93eb05cefc5f"} Sep 11 11:43:02.700974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:02.700943 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" event={"ID":"5034794a-d3e7-4a1f-b7a5-8f23f9817239","Type":"ContainerStarted","Data":"0fa8e3db6eb9ee2f3a1eab552545460ab587ff42a74fcb771f7303ecfda0f2f7"} Sep 11 11:43:02.701122 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:02.701078 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:02.734515 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:02.734465 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" podStartSLOduration=1.9153694799999998 podStartE2EDuration="2.734449064s" totalImagesPullingTime="819.079584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:43:02.732410869 +0000 UTC m=+544.367004005" watchObservedRunningTime="2026-09-11 11:43:02.734449064 +0000 UTC m=+544.369042230" Sep 11 11:43:04.716632 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:04.716601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8x8wg" event={"ID":"60654a9b-1719-4c21-aae8-a91c831bd3b0","Type":"ContainerStarted","Data":"7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d"} Sep 11 11:43:04.747554 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:04.747514 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-8x8wg" podStartSLOduration=1.848649202 podStartE2EDuration="3.747499594s" totalImagesPullingTime="1.898850392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:43:04.747323697 +0000 UTC m=+546.381916842" watchObservedRunningTime="2026-09-11 11:43:04.747499594 +0000 UTC m=+546.382092726" Sep 11 11:43:13.710837 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:13.710807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8cltk" Sep 11 11:43:39.832731 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:39.832699 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8x8wg"] Sep 11 11:43:39.833136 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:39.832864 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-8x8wg" podUID="60654a9b-1719-4c21-aae8-a91c831bd3b0" containerName="authorino" containerID="cri-o://7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d" gracePeriod=30 Sep 11 11:43:40.086958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.086901 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8x8wg" Sep 11 11:43:40.148238 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.148209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz9tr\" (UniqueName: \"kubernetes.io/projected/60654a9b-1719-4c21-aae8-a91c831bd3b0-kube-api-access-rz9tr\") pod \"60654a9b-1719-4c21-aae8-a91c831bd3b0\" (UID: \"60654a9b-1719-4c21-aae8-a91c831bd3b0\") " Sep 11 11:43:40.150316 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.150296 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60654a9b-1719-4c21-aae8-a91c831bd3b0-kube-api-access-rz9tr" (OuterVolumeSpecName: "kube-api-access-rz9tr") pod "60654a9b-1719-4c21-aae8-a91c831bd3b0" (UID: "60654a9b-1719-4c21-aae8-a91c831bd3b0"). InnerVolumeSpecName "kube-api-access-rz9tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:43:40.248948 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.248914 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9tr\" (UniqueName: \"kubernetes.io/projected/60654a9b-1719-4c21-aae8-a91c831bd3b0-kube-api-access-rz9tr\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:43:40.905742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.905706 2580 generic.go:356] "Generic (PLEG): container finished" podID="60654a9b-1719-4c21-aae8-a91c831bd3b0" containerID="7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d" exitCode=0 Sep 11 11:43:40.905742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.905739 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8x8wg" Sep 11 11:43:40.906158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.905750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8x8wg" event={"ID":"60654a9b-1719-4c21-aae8-a91c831bd3b0","Type":"ContainerDied","Data":"7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d"} Sep 11 11:43:40.906158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.905786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8x8wg" event={"ID":"60654a9b-1719-4c21-aae8-a91c831bd3b0","Type":"ContainerDied","Data":"1023e6d7f67b2ddd3173087a191ebc8a8613d5e386e41101bc5b93eb05cefc5f"} Sep 11 11:43:40.906158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.905805 2580 scope.go:117] "RemoveContainer" containerID="7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d" Sep 11 11:43:40.916907 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.916888 2580 scope.go:117] "RemoveContainer" containerID="7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d" Sep 11 11:43:40.917248 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:43:40.917218 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d\": container with ID starting with 7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d not found: ID does not exist" containerID="7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d" Sep 11 11:43:40.917342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.917252 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d"} err="failed to get container status \"7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d\": rpc error: code = NotFound desc = could not find container \"7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d\": container with ID starting with 7b13c89a0c1ef8676dc8ecff54a96b90bf494df064a37e2e4dfe4b1cec87424d not found: ID does not exist" Sep 11 11:43:40.937724 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.937693 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8x8wg"] Sep 11 11:43:40.943200 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:40.943182 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8x8wg"] Sep 11 11:43:42.872916 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:42.872883 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60654a9b-1719-4c21-aae8-a91c831bd3b0" path="/var/lib/kubelet/pods/60654a9b-1719-4c21-aae8-a91c831bd3b0/volumes" Sep 11 11:43:58.819111 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:58.819086 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:43:58.819682 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:43:58.819663 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:44:24.486474 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:24.486393 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:44:34.201372 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:34.201338 2580 generic.go:356] "Generic (PLEG): container finished" podID="0233a32a-dd49-4369-974e-3e82d7103a23" containerID="d052f732901e4268b8383e411545cf492e0c12f0e3346eb9b140aabd0b2b5668" exitCode=0 Sep 11 11:44:34.201794 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:34.201410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" event={"ID":"0233a32a-dd49-4369-974e-3e82d7103a23","Type":"ContainerDied","Data":"d052f732901e4268b8383e411545cf492e0c12f0e3346eb9b140aabd0b2b5668"} Sep 11 11:44:34.201836 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:34.201799 2580 scope.go:117] "RemoveContainer" containerID="d052f732901e4268b8383e411545cf492e0c12f0e3346eb9b140aabd0b2b5668" Sep 11 11:44:34.202775 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:34.202746 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:44:34.202842 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:34.202812 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:44:34.202886 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:34.202841 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:44:35.208612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:35.208580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" event={"ID":"0233a32a-dd49-4369-974e-3e82d7103a23","Type":"ContainerStarted","Data":"16fb674ed914fa680291ec58eedf952fb084113a5da31b5969839846305cc944"} Sep 11 11:44:35.307093 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:35.307060 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:44:35.307271 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:35.307248 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-l8td8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 11:44:35.307331 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:35.307308 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" podUID="0233a32a-dd49-4369-974e-3e82d7103a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 11:44:36.307678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:36.307639 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-l8td8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 11:44:36.308064 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:36.307702 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" podUID="0233a32a-dd49-4369-974e-3e82d7103a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 11:44:37.307043 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:37.307008 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-l8td8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 11:44:37.307213 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:37.307090 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" podUID="0233a32a-dd49-4369-974e-3e82d7103a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 11:44:38.307329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:38.307298 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-l8td8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 11:44:38.307700 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:38.307350 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" podUID="0233a32a-dd49-4369-974e-3e82d7103a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 11:44:39.307258 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:39.307225 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-l8td8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 11:44:39.307473 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:39.307290 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" podUID="0233a32a-dd49-4369-974e-3e82d7103a23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 11:44:40.311313 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:40.311287 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:44:40.311685 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:40.311542 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:44:40.312130 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:40.312114 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-l8td8" Sep 11 11:44:51.298000 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:44:51.297970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:45:03.176450 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.176392 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-d692g"] Sep 11 11:45:03.177002 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.176899 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60654a9b-1719-4c21-aae8-a91c831bd3b0" containerName="authorino" Sep 11 11:45:03.177002 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.176916 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60654a9b-1719-4c21-aae8-a91c831bd3b0" containerName="authorino" Sep 11 11:45:03.177002 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.176990 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60654a9b-1719-4c21-aae8-a91c831bd3b0" containerName="authorino" Sep 11 11:45:03.179170 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.179154 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.184311 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.184288 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 11:45:03.184544 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.184525 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7cwhz\"" Sep 11 11:45:03.185064 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.185023 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 11 11:45:03.185753 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.185393 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 11:45:03.212435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.212398 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-d692g"] Sep 11 11:45:03.346535 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.346490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce38404-0099-44ca-9f81-89b91ce2c160-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.346535 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.346525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tht8g\" (UniqueName: \"kubernetes.io/projected/dce38404-0099-44ca-9f81-89b91ce2c160-kube-api-access-tht8g\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.346805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.346572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.346805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.346629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.346805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.346703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.346805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.346722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.447933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce38404-0099-44ca-9f81-89b91ce2c160-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tht8g\" (UniqueName: \"kubernetes.io/projected/dce38404-0099-44ca-9f81-89b91ce2c160-kube-api-access-tht8g\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448454 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448454 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.448575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.448505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.450488 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.450468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce38404-0099-44ca-9f81-89b91ce2c160-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.450769 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.450742 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce38404-0099-44ca-9f81-89b91ce2c160-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.473313 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.473277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tht8g\" (UniqueName: \"kubernetes.io/projected/dce38404-0099-44ca-9f81-89b91ce2c160-kube-api-access-tht8g\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d692g\" (UID: \"dce38404-0099-44ca-9f81-89b91ce2c160\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.489459 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.489434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:03.776229 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.776207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-d692g"] Sep 11 11:45:03.778250 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:45:03.778227 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce38404_0099_44ca_9f81_89b91ce2c160.slice/crio-fedd9bcfb20b03accdd40289346047146161895d4b8e637f939f93f92d37819f WatchSource:0}: Error finding container fedd9bcfb20b03accdd40289346047146161895d4b8e637f939f93f92d37819f: Status 404 returned error can't find the container with id fedd9bcfb20b03accdd40289346047146161895d4b8e637f939f93f92d37819f Sep 11 11:45:03.779463 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:03.779448 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:45:04.070701 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:04.070608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:45:04.364659 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:04.364503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" event={"ID":"dce38404-0099-44ca-9f81-89b91ce2c160","Type":"ContainerStarted","Data":"fedd9bcfb20b03accdd40289346047146161895d4b8e637f939f93f92d37819f"} Sep 11 11:45:05.372075 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:05.371969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" event={"ID":"dce38404-0099-44ca-9f81-89b91ce2c160","Type":"ContainerStarted","Data":"4435e44f33a99cc8a3969eed233d230fc1b184dc302635d0f8bf7efb51829336"} Sep 11 11:45:05.403428 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:05.403374 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" podStartSLOduration=1.264051701 podStartE2EDuration="2.403357377s" totalImagesPullingTime="1.139305676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:45:05.401322597 +0000 UTC m=+667.035915732" watchObservedRunningTime="2026-09-11 11:45:05.403357377 +0000 UTC m=+667.037950508" Sep 11 11:45:13.490204 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:13.490173 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:13.490712 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:13.490222 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:13.502798 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:13.502778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:14.429648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:14.429622 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d692g" Sep 11 11:45:20.619470 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:20.619441 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:45:34.783086 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.783029 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh"] Sep 11 11:45:34.787194 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.787177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.792162 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.792142 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 11:45:34.816192 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.816173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.816304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.816206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5c4\" (UniqueName: \"kubernetes.io/projected/cd736a12-ce18-47ee-8634-c8f62988ecf9-kube-api-access-6f5c4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.816304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.816237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd736a12-ce18-47ee-8634-c8f62988ecf9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.816304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.816264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.816304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.816293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.816452 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.816306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.826371 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.826348 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh"] Sep 11 11:45:34.917248 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917438 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5c4\" (UniqueName: \"kubernetes.io/projected/cd736a12-ce18-47ee-8634-c8f62988ecf9-kube-api-access-6f5c4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917438 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd736a12-ce18-47ee-8634-c8f62988ecf9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917438 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917615 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917615 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917721 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917721 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.917852 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.917834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.919720 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.919700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd736a12-ce18-47ee-8634-c8f62988ecf9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.919891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.919871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd736a12-ce18-47ee-8634-c8f62988ecf9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:34.932385 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:34.932351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5c4\" (UniqueName: \"kubernetes.io/projected/cd736a12-ce18-47ee-8634-c8f62988ecf9-kube-api-access-6f5c4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-d8mwh\" (UID: \"cd736a12-ce18-47ee-8634-c8f62988ecf9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:35.097839 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:35.097754 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:35.389657 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:35.389631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh"] Sep 11 11:45:35.390875 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:45:35.390857 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd736a12_ce18_47ee_8634_c8f62988ecf9.slice/crio-7c90bab642e0b89f8aa09a519784309f8ca23c6c01b5779f63a5889d3a0dfb7b WatchSource:0}: Error finding container 7c90bab642e0b89f8aa09a519784309f8ca23c6c01b5779f63a5889d3a0dfb7b: Status 404 returned error can't find the container with id 7c90bab642e0b89f8aa09a519784309f8ca23c6c01b5779f63a5889d3a0dfb7b Sep 11 11:45:35.533291 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:35.533255 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" event={"ID":"cd736a12-ce18-47ee-8634-c8f62988ecf9","Type":"ContainerStarted","Data":"7c90bab642e0b89f8aa09a519784309f8ca23c6c01b5779f63a5889d3a0dfb7b"} Sep 11 11:45:36.307290 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:36.307257 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:45:36.542617 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:36.542572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" event={"ID":"cd736a12-ce18-47ee-8634-c8f62988ecf9","Type":"ContainerStarted","Data":"0d49b07355a0a803eafa7fe255ddbf75f66214d0872017559d5c366b509aee6f"} Sep 11 11:45:36.627639 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:36.627538 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" podStartSLOduration=2.425910184 podStartE2EDuration="2.627521912s" totalImagesPullingTime="201.611728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:45:36.622806881 +0000 UTC m=+698.257400014" watchObservedRunningTime="2026-09-11 11:45:36.627521912 +0000 UTC m=+698.262115036" Sep 11 11:45:45.098436 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:45.098404 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:45.098920 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:45.098465 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:45.110891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:45.110868 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:45.602939 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:45.602912 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-d8mwh" Sep 11 11:45:48.511359 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:45:48.511331 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:46:01.455023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:01.454942 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:46:09.494624 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.494584 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7"] Sep 11 11:46:09.498618 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.498594 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.504118 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.504088 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 11 11:46:09.522766 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.522737 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7"] Sep 11 11:46:09.618241 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.618206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.618241 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.618238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6744\" (UniqueName: \"kubernetes.io/projected/dddc057a-c848-4d9f-9ee9-649f3a695755-kube-api-access-k6744\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.618459 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.618265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.618459 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.618314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.618459 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.618383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddc057a-c848-4d9f-9ee9-649f3a695755-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.618459 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.618441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719289 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6744\" (UniqueName: \"kubernetes.io/projected/dddc057a-c848-4d9f-9ee9-649f3a695755-kube-api-access-k6744\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddc057a-c848-4d9f-9ee9-649f3a695755-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719752 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719729 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.719880 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.719864 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.721843 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.721820 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dddc057a-c848-4d9f-9ee9-649f3a695755-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.722122 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.722104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dddc057a-c848-4d9f-9ee9-649f3a695755-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.734612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.734587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6744\" (UniqueName: \"kubernetes.io/projected/dddc057a-c848-4d9f-9ee9-649f3a695755-kube-api-access-k6744\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7\" (UID: \"dddc057a-c848-4d9f-9ee9-649f3a695755\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:09.807785 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:09.807694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:10.091596 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:10.091569 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7"] Sep 11 11:46:10.092620 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:46:10.092597 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddddc057a_c848_4d9f_9ee9_649f3a695755.slice/crio-9d0da98fd2418b367d91aa7401f4555ec682b55475d0b77b71a7eeba43519da4 WatchSource:0}: Error finding container 9d0da98fd2418b367d91aa7401f4555ec682b55475d0b77b71a7eeba43519da4: Status 404 returned error can't find the container with id 9d0da98fd2418b367d91aa7401f4555ec682b55475d0b77b71a7eeba43519da4 Sep 11 11:46:10.732653 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:10.732618 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" event={"ID":"dddc057a-c848-4d9f-9ee9-649f3a695755","Type":"ContainerStarted","Data":"6d7044a11718a23f4ee64ba588d7ba642a5ccbf4273cf3dface5304f7095a98b"} Sep 11 11:46:10.733059 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:10.732663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" event={"ID":"dddc057a-c848-4d9f-9ee9-649f3a695755","Type":"ContainerStarted","Data":"9d0da98fd2418b367d91aa7401f4555ec682b55475d0b77b71a7eeba43519da4"} Sep 11 11:46:10.774968 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:10.774908 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" podStartSLOduration=1.573020205 podStartE2EDuration="1.77489099s" totalImagesPullingTime="201.870785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:46:10.77451942 +0000 UTC m=+732.409112553" watchObservedRunningTime="2026-09-11 11:46:10.77489099 +0000 UTC m=+732.409484124" Sep 11 11:46:11.756834 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:11.756802 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:46:19.808164 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:19.808113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:19.808643 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:19.808186 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:19.821134 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:19.821103 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:46:20.798430 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:46:20.798396 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7" Sep 11 11:47:34.245708 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.245611 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q"] Sep 11 11:47:34.249129 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.249110 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.252273 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.252251 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dbe0e7-openshift-default-dockercfg-wmsbf\"" Sep 11 11:47:34.275321 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.275290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q"] Sep 11 11:47:34.373340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istiod-ca-cert\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-envoy\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373551 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-certs\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373551 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-socket\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373551 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-podinfo\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373551 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373537 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljwrg\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-kube-api-access-ljwrg\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-credential-socket\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-token\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.373695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.373668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-data\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.474990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.474946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-data\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istiod-ca-cert\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-envoy\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475179 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-certs\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475346 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-socket\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475346 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-podinfo\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475346 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljwrg\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-kube-api-access-ljwrg\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475676 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-credential-socket\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475676 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-token\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475676 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-data\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.475963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.475934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istiod-ca-cert\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.476108 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.476068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-certs\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.476261 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.476241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-credential-socket\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.476598 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.476577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-socket\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.478641 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.478606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-envoy\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.478990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.478966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-podinfo\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.483907 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.483873 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-token\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.484319 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.484300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljwrg\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-kube-api-access-ljwrg\") pod \"e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.560421 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.560339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:34.846998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.846868 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q"] Sep 11 11:47:34.849833 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:47:34.849809 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c8adfc_0d16_4e57_be2c_2d0f2d690eee.slice/crio-0b542565d6dd8e5d132d898d137bba9ed736c70e0aebd6c7a2e942e501dc4757 WatchSource:0}: Error finding container 0b542565d6dd8e5d132d898d137bba9ed736c70e0aebd6c7a2e942e501dc4757: Status 404 returned error can't find the container with id 0b542565d6dd8e5d132d898d137bba9ed736c70e0aebd6c7a2e942e501dc4757 Sep 11 11:47:34.851943 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.851911 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:34.852055 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.852003 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:34.852125 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:34.852100 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:35.184598 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:35.184565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" event={"ID":"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee","Type":"ContainerStarted","Data":"dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505"} Sep 11 11:47:35.184786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:35.184622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" event={"ID":"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee","Type":"ContainerStarted","Data":"0b542565d6dd8e5d132d898d137bba9ed736c70e0aebd6c7a2e942e501dc4757"} Sep 11 11:47:35.215254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:35.215189 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podStartSLOduration=1.215169143 podStartE2EDuration="1.215169143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:47:35.213363874 +0000 UTC m=+816.847957022" watchObservedRunningTime="2026-09-11 11:47:35.215169143 +0000 UTC m=+816.849762277" Sep 11 11:47:35.561197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:35.561095 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:35.566899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:35.566872 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:36.190318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:36.190289 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:36.191291 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:36.191264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:37.514513 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:37.514482 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:47:42.406901 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.406869 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g"] Sep 11 11:47:42.410998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.410975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.413516 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.413497 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bddd3ec1-openshift-default-dockercfg-t7d5j\"" Sep 11 11:47:42.445308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.445282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g"] Sep 11 11:47:42.551235 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-envoy\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-podinfo\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-data\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istiod-ca-cert\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79z5\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-kube-api-access-d79z5\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-socket\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-certs\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551664 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-credential-socket\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.551664 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.551480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-token\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.652764 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.652733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-credential-socket\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.652974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.652775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-token\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.652974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.652905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-envoy\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.652974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.652938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-podinfo\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.652974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.652958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-data\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.652974 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.652975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istiod-ca-cert\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.652990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79z5\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-kube-api-access-d79z5\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.653249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-credential-socket\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653375 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.653357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-data\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653415 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.653398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-socket\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653469 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.653427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-certs\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.653619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-socket\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653741 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.653721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-certs\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.653786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.653746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istiod-ca-cert\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.655269 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.655242 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-envoy\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.655599 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.655577 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-podinfo\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.673723 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.673667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-token\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.674287 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.674268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79z5\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-kube-api-access-d79z5\") pod \"e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:42.721596 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:42.721564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:43.022294 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:47:43.022254 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548f5d51_cbf1_4553_aec3_cfc0de48fcae.slice/crio-d9142dcf3b75ad2faa286cc49d6e9edc78d0befddd9a08f5e8bca70ff179d363 WatchSource:0}: Error finding container d9142dcf3b75ad2faa286cc49d6e9edc78d0befddd9a08f5e8bca70ff179d363: Status 404 returned error can't find the container with id d9142dcf3b75ad2faa286cc49d6e9edc78d0befddd9a08f5e8bca70ff179d363 Sep 11 11:47:43.022421 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.022318 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g"] Sep 11 11:47:43.024605 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.024570 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:43.024685 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.024657 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:43.024724 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.024704 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:43.231682 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.231643 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" event={"ID":"548f5d51-cbf1-4553-aec3-cfc0de48fcae","Type":"ContainerStarted","Data":"8c01517e96ca1d17715b14042565998a7b9b43cfd3c42803a0be7be91c5edded"} Sep 11 11:47:43.231866 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.231698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" event={"ID":"548f5d51-cbf1-4553-aec3-cfc0de48fcae","Type":"ContainerStarted","Data":"d9142dcf3b75ad2faa286cc49d6e9edc78d0befddd9a08f5e8bca70ff179d363"} Sep 11 11:47:43.255270 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.255212 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podStartSLOduration=1.255191156 podStartE2EDuration="1.255191156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:47:43.251196649 +0000 UTC m=+824.885789792" watchObservedRunningTime="2026-09-11 11:47:43.255191156 +0000 UTC m=+824.889784290" Sep 11 11:47:43.575320 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.575281 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6"] Sep 11 11:47:43.583059 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.581302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.592058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.589588 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6"] Sep 11 11:47:43.592058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.591329 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5a5ab7-openshift-default-dockercfg-nhq5h\"" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8krj\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-kube-api-access-j8krj\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-envoy\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-certs\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-credential-socket\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-token\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-podinfo\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istiod-ca-cert\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-socket\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.671076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.670890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-data\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.722233 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.722189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:43.771582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8krj\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-kube-api-access-j8krj\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.771786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-envoy\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.771786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-certs\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.771786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-credential-socket\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.771786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-token\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.771786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-podinfo\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.772086 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istiod-ca-cert\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.772086 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-socket\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.772086 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.771858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-data\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.772381 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.772353 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-data\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.773748 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.773712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istiod-ca-cert\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.774560 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.774235 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-socket\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.774560 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.774517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-certs\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.774742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.774609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-credential-socket\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.785866 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.785810 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8krj\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-kube-api-access-j8krj\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.786279 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.786257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-envoy\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.788716 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.788674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-token\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.789318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.789296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-podinfo\") pod \"e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:43.931216 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:43.931184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:44.238244 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:44.238219 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6"] Sep 11 11:47:44.239573 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:47:44.239549 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4ae0e2_119c_4f79_aedf_54c9f1ac471d.slice/crio-1e99b5f75bf1d76b1cb78888069e49fb8a270bcd340ee9a44b77a1e62d23620e WatchSource:0}: Error finding container 1e99b5f75bf1d76b1cb78888069e49fb8a270bcd340ee9a44b77a1e62d23620e: Status 404 returned error can't find the container with id 1e99b5f75bf1d76b1cb78888069e49fb8a270bcd340ee9a44b77a1e62d23620e Sep 11 11:47:44.241695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:44.241661 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:44.241801 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:44.241742 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:44.241801 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:44.241787 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:44.727518 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:44.727492 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:45.245319 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:45.245286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" event={"ID":"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d","Type":"ContainerStarted","Data":"de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0"} Sep 11 11:47:45.245486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:45.245337 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" event={"ID":"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d","Type":"ContainerStarted","Data":"1e99b5f75bf1d76b1cb78888069e49fb8a270bcd340ee9a44b77a1e62d23620e"} Sep 11 11:47:45.245537 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:45.245518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:45.246650 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:45.246628 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:45.269058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:45.266387 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podStartSLOduration=2.266368275 podStartE2EDuration="2.266368275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:47:45.265497093 +0000 UTC m=+826.900090228" watchObservedRunningTime="2026-09-11 11:47:45.266368275 +0000 UTC m=+826.900961409" Sep 11 11:47:45.931686 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:45.931651 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:45.936803 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:45.936778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:46.251116 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:46.250996 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:46.252004 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:46.251987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:47:47.935000 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:47.934967 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:47:52.092177 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.090921 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:34562->10.132.0.50:15021: read: connection reset by peer" start-of-body= Sep 11 11:47:52.092177 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.091011 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": read tcp 10.132.0.2:34562->10.132.0.50:15021: read: connection reset by peer" Sep 11 11:47:52.297356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.297280 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerID="dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505" exitCode=0 Sep 11 11:47:52.297356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.297343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" event={"ID":"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee","Type":"ContainerDied","Data":"dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505"} Sep 11 11:47:52.297751 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.297737 2580 scope.go:117] "RemoveContainer" containerID="dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505" Sep 11 11:47:52.298809 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.298779 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:52.298861 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.298852 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:52.298895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:52.298884 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:53.310130 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:53.310089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" event={"ID":"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee","Type":"ContainerStarted","Data":"d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2"} Sep 11 11:47:53.561333 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:53.560503 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:53.561333 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:53.560939 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 11:47:53.561333 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:53.560988 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 11:47:54.561511 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:54.561031 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 11:47:54.561511 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:54.561117 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 11:47:55.562146 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:55.561626 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 11:47:55.562146 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:55.561690 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 11:47:56.561499 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:56.561028 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 11:47:56.561499 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:56.561122 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 11:47:57.561796 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:57.561749 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 11:47:57.562337 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:57.561827 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 11:47:58.353583 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.353556 2580 generic.go:356] "Generic (PLEG): container finished" podID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerID="8c01517e96ca1d17715b14042565998a7b9b43cfd3c42803a0be7be91c5edded" exitCode=0 Sep 11 11:47:58.353786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.353626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" event={"ID":"548f5d51-cbf1-4553-aec3-cfc0de48fcae","Type":"ContainerDied","Data":"8c01517e96ca1d17715b14042565998a7b9b43cfd3c42803a0be7be91c5edded"} Sep 11 11:47:58.354261 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.354247 2580 scope.go:117] "RemoveContainer" containerID="8c01517e96ca1d17715b14042565998a7b9b43cfd3c42803a0be7be91c5edded" Sep 11 11:47:58.355509 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.355478 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:58.355588 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.355544 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:58.355588 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.355573 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:47:58.564465 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.564409 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Sep 11 11:47:58.564963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:58.564473 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Sep 11 11:47:59.362061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:59.361955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" event={"ID":"548f5d51-cbf1-4553-aec3-cfc0de48fcae","Type":"ContainerStarted","Data":"00e75e8467b2307ec0144da0bf1acb24d6e0b2b5f15cd2da443996363b1e59aa"} Sep 11 11:47:59.580715 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:59.580675 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:59.581529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:59.581503 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:59.582627 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:59.582570 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:47:59.722094 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:59.722064 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:47:59.723013 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:59.722985 2580 patch_prober.go:28] interesting pod/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 11:47:59.723111 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:47:59.723068 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 11:48:00.246250 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:00.246216 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:48:00.722394 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:00.722357 2580 patch_prober.go:28] interesting pod/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 11:48:00.722783 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:00.722413 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 11:48:01.722675 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:01.722637 2580 patch_prober.go:28] interesting pod/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 11:48:01.723122 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:01.722694 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 11:48:02.723091 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.723030 2580 patch_prober.go:28] interesting pod/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 11:48:02.723485 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.723116 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 11:48:02.744192 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.744163 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw"] Sep 11 11:48:02.746991 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.746971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.749338 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.749316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c6bcf5-openshift-default-dockercfg-v4j5t\"" Sep 11 11:48:02.780062 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.776696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw"] Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-socket\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2852r\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-kube-api-access-2852r\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798632 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-envoy\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-istio-token\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-certs\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4133241f-99b7-481f-9977-925126d8407b-istio-podinfo\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-credential-socket\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-data\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.802058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.798921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4133241f-99b7-481f-9977-925126d8407b-istiod-ca-cert\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.899549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.899508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-credential-socket\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.899837 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.899815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-data\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.899969 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.899954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4133241f-99b7-481f-9977-925126d8407b-istiod-ca-cert\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.900105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.900089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-socket\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.900222 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.900209 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2852r\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-kube-api-access-2852r\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.900349 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.900336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-envoy\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.900505 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.900488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-istio-token\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.900628 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.900614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-certs\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.900755 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.900742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4133241f-99b7-481f-9977-925126d8407b-istio-podinfo\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.903990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.901653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-socket\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.903990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.901904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-credential-socket\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.903990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.902162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-data\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.903990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.902784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4133241f-99b7-481f-9977-925126d8407b-istiod-ca-cert\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.903990 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.903788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-certs\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.908596 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.908564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-envoy\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.909438 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.909418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4133241f-99b7-481f-9977-925126d8407b-istio-podinfo\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.915812 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.915721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-istio-token\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:02.919634 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:02.919602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2852r\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-kube-api-access-2852r\") pod \"e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:03.058700 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.058385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:03.386885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.386836 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw"] Sep 11 11:48:03.393317 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:48:03.393285 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4133241f_99b7_481f_9977_925126d8407b.slice/crio-b6959762a4296f60ae8102b83c368b9d7d30854ab7f4662b6e59dd78a3579575 WatchSource:0}: Error finding container b6959762a4296f60ae8102b83c368b9d7d30854ab7f4662b6e59dd78a3579575: Status 404 returned error can't find the container with id b6959762a4296f60ae8102b83c368b9d7d30854ab7f4662b6e59dd78a3579575 Sep 11 11:48:03.396185 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.396151 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:03.396316 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.396233 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:03.396316 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.396278 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:03.692843 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.692803 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:48:03.722609 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.722566 2580 patch_prober.go:28] interesting pod/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 11:48:03.722766 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:03.722647 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 11:48:04.408095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:04.408030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" event={"ID":"4133241f-99b7-481f-9977-925126d8407b","Type":"ContainerStarted","Data":"d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a"} Sep 11 11:48:04.408525 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:04.408119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" event={"ID":"4133241f-99b7-481f-9977-925126d8407b","Type":"ContainerStarted","Data":"b6959762a4296f60ae8102b83c368b9d7d30854ab7f4662b6e59dd78a3579575"} Sep 11 11:48:04.429673 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:04.429621 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podStartSLOduration=2.42960199 podStartE2EDuration="2.42960199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:04.427269897 +0000 UTC m=+846.061863034" watchObservedRunningTime="2026-09-11 11:48:04.42960199 +0000 UTC m=+846.064195122" Sep 11 11:48:04.722790 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:04.722701 2580 patch_prober.go:28] interesting pod/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 11:48:04.722790 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:04.722773 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 11:48:05.059260 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:05.059181 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:05.064992 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:05.064966 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:05.413720 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:05.413677 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:05.414840 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:05.414817 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:05.727512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:05.727435 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:48:05.728627 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:05.728608 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:48:07.427253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:07.427219 2580 generic.go:356] "Generic (PLEG): container finished" podID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerID="de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0" exitCode=0 Sep 11 11:48:07.427824 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:07.427287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" event={"ID":"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d","Type":"ContainerDied","Data":"de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0"} Sep 11 11:48:07.427824 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:07.427785 2580 scope.go:117] "RemoveContainer" containerID="de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0" Sep 11 11:48:07.429131 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:07.429099 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:07.429196 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:07.429183 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:07.429232 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:07.429221 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:08.435363 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:08.435329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" event={"ID":"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d","Type":"ContainerStarted","Data":"48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82"} Sep 11 11:48:08.931604 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:08.931571 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:48:08.931812 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:08.931787 2580 patch_prober.go:28] interesting pod/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 11:48:08.931869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:08.931849 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 11:48:09.931986 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:09.931949 2580 patch_prober.go:28] interesting pod/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 11:48:09.932449 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:09.932003 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 11:48:10.932126 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:10.932093 2580 patch_prober.go:28] interesting pod/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 11:48:10.932536 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:10.932156 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 11:48:11.932234 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:11.932193 2580 patch_prober.go:28] interesting pod/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 11:48:11.932620 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:11.932271 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 11:48:12.931565 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:12.931528 2580 patch_prober.go:28] interesting pod/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 11:48:12.931798 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:12.931609 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 11:48:13.809767 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:13.809726 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:48:13.931725 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:13.931687 2580 patch_prober.go:28] interesting pod/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 11:48:13.931910 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:13.931760 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 11:48:14.956223 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:14.956186 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:48:14.956708 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:14.956685 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:48:14.957574 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:14.957556 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:48:21.455269 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:21.455216 2580 patch_prober.go:28] interesting pod/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 11:48:21.455782 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:21.455305 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 11:48:22.189295 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.189259 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2"] Sep 11 11:48:22.192268 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.192252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.196419 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.196400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-31a432b7-kserve-self-signed-certs\"" Sep 11 11:48:22.223237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.223206 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2"] Sep 11 11:48:22.332142 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.332101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-model-cache\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.332332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.332159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tmp-dir\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.332332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.332267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjrt\" (UniqueName: \"kubernetes.io/projected/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-kube-api-access-fkjrt\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.332332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.332334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-dshm\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.332507 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.332381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-home\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.332507 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.332410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tls-certs\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.433682 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.433649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tmp-dir\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.433879 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.433716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjrt\" (UniqueName: \"kubernetes.io/projected/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-kube-api-access-fkjrt\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.433879 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.433761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-dshm\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.433879 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.433789 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-home\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.434078 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.433881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tls-certs\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.434078 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.433951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-model-cache\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.434219 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.434193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-home\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.434339 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.434322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tmp-dir\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.434401 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.434348 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-model-cache\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.436288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.436257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-dshm\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.436579 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.436559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tls-certs\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.443959 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.443898 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjrt\" (UniqueName: \"kubernetes.io/projected/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-kube-api-access-fkjrt\") pod \"e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.503828 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.503784 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:22.794779 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:22.794746 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2"] Sep 11 11:48:22.796424 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:48:22.796397 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f4d4e75_0326_497e_92c8_5f9fd4e4c5f5.slice/crio-ea3d22b698e89375065a824f7aa3525f48a3a64987491b77dab24afe832ec559 WatchSource:0}: Error finding container ea3d22b698e89375065a824f7aa3525f48a3a64987491b77dab24afe832ec559: Status 404 returned error can't find the container with id ea3d22b698e89375065a824f7aa3525f48a3a64987491b77dab24afe832ec559 Sep 11 11:48:23.536504 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.536478 2580 generic.go:356] "Generic (PLEG): container finished" podID="4133241f-99b7-481f-9977-925126d8407b" containerID="d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a" exitCode=0 Sep 11 11:48:23.536896 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.536558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" event={"ID":"4133241f-99b7-481f-9977-925126d8407b","Type":"ContainerDied","Data":"d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a"} Sep 11 11:48:23.537070 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.537028 2580 scope.go:117] "RemoveContainer" containerID="d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a" Sep 11 11:48:23.538249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.538218 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:23.538334 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.538288 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:23.538334 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.538318 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:23.538412 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.538315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" event={"ID":"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5","Type":"ContainerStarted","Data":"b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f"} Sep 11 11:48:23.538446 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.538425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" event={"ID":"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5","Type":"ContainerStarted","Data":"ea3d22b698e89375065a824f7aa3525f48a3a64987491b77dab24afe832ec559"} Sep 11 11:48:23.577935 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:23.577889 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" podStartSLOduration=1.577870101 podStartE2EDuration="1.577870101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:23.574874736 +0000 UTC m=+865.209467870" watchObservedRunningTime="2026-09-11 11:48:23.577870101 +0000 UTC m=+865.212463235" Sep 11 11:48:24.549022 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:24.548985 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" event={"ID":"4133241f-99b7-481f-9977-925126d8407b","Type":"ContainerStarted","Data":"d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6"} Sep 11 11:48:25.058950 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:25.058910 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:25.059219 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:25.059196 2580 patch_prober.go:28] interesting pod/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 11:48:25.059300 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:25.059256 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 11:48:26.059357 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:26.059321 2580 patch_prober.go:28] interesting pod/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 11:48:26.059744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:26.059377 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 11:48:27.059491 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:27.059449 2580 patch_prober.go:28] interesting pod/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 11:48:27.060095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:27.060063 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 11:48:27.324368 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:27.324289 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g"] Sep 11 11:48:27.324534 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:27.324502 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" containerID="cri-o://00e75e8467b2307ec0144da0bf1acb24d6e0b2b5f15cd2da443996363b1e59aa" gracePeriod=30 Sep 11 11:48:28.059385 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.059346 2580 patch_prober.go:28] interesting pod/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 11:48:28.059560 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.059408 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 11:48:28.289062 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.288665 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt"] Sep 11 11:48:28.297179 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.297155 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.299944 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.299925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5fdde1a2-gw-openshift-default-dockercfg-5sn76\"" Sep 11 11:48:28.305550 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.305527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt"] Sep 11 11:48:28.403996 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.403962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-token\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-socket\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-credential-socket\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-data\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404345 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-podinfo\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404345 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-envoy\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404345 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istiod-ca-cert\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404345 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-certs\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.404495 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.404386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlm88\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-kube-api-access-zlm88\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.505796 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.505754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-token\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.505985 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.505822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-socket\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.505985 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.505886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-credential-socket\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.505985 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.505920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-data\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.506195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.505983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-podinfo\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.506195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.506054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-envoy\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.506195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.506090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istiod-ca-cert\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.506195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.506120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-certs\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.506195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.506173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlm88\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-kube-api-access-zlm88\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.507057 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.506985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-credential-socket\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.507307 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.507280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-certs\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.507989 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.507872 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-data\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.508571 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.508512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-socket\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.509390 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.509366 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istiod-ca-cert\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.510294 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.510250 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-envoy\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.512526 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.512495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-podinfo\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.520416 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.520398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlm88\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-kube-api-access-zlm88\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.520573 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.520440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-token\") pod \"e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.611253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.611224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:28.913931 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.913905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt"] Sep 11 11:48:28.918382 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.918349 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:28.918472 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.918411 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:28.918472 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:28.918442 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:29.059664 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:29.059632 2580 patch_prober.go:28] interesting pod/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 11:48:29.060131 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:29.059696 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 11:48:29.457235 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:29.457201 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:48:29.587227 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:29.587193 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" event={"ID":"efe3f2a7-62b7-4da8-a0c4-895474b923e9","Type":"ContainerStarted","Data":"bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51"} Sep 11 11:48:29.587395 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:29.587246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" event={"ID":"efe3f2a7-62b7-4da8-a0c4-895474b923e9","Type":"ContainerStarted","Data":"a6dd270cda2dfe64a1a567c1de7a656a233a4bee570081fc3f88b55fbe1df071"} Sep 11 11:48:29.612267 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:29.612239 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:29.656886 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:29.656833 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podStartSLOduration=1.6568176449999998 podStartE2EDuration="1.656817645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:48:29.652232647 +0000 UTC m=+871.286825781" watchObservedRunningTime="2026-09-11 11:48:29.656817645 +0000 UTC m=+871.291410777" Sep 11 11:48:30.062433 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:30.062405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:30.062892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:30.062793 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:30.063555 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:30.063533 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:48:30.617475 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:30.617443 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:31.598929 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:31.598904 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:31.599938 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:31.599919 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:32.504424 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.504393 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:32.504575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.504451 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:32.522254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.521939 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:32.606507 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.606481 2580 generic.go:356] "Generic (PLEG): container finished" podID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerID="00e75e8467b2307ec0144da0bf1acb24d6e0b2b5f15cd2da443996363b1e59aa" exitCode=0 Sep 11 11:48:32.606887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.606554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" event={"ID":"548f5d51-cbf1-4553-aec3-cfc0de48fcae","Type":"ContainerDied","Data":"00e75e8467b2307ec0144da0bf1acb24d6e0b2b5f15cd2da443996363b1e59aa"} Sep 11 11:48:32.606887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.606613 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" event={"ID":"548f5d51-cbf1-4553-aec3-cfc0de48fcae","Type":"ContainerDied","Data":"d9142dcf3b75ad2faa286cc49d6e9edc78d0befddd9a08f5e8bca70ff179d363"} Sep 11 11:48:32.606887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.606638 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9142dcf3b75ad2faa286cc49d6e9edc78d0befddd9a08f5e8bca70ff179d363" Sep 11 11:48:32.606887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.606655 2580 scope.go:117] "RemoveContainer" containerID="8c01517e96ca1d17715b14042565998a7b9b43cfd3c42803a0be7be91c5edded" Sep 11 11:48:32.610246 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.610229 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:48:32.618612 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.618592 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:48:32.648935 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.648905 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-token\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649085 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.648953 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-socket\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649085 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649006 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-envoy\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649085 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649048 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-podinfo\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649085 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649074 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d79z5\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-kube-api-access-d79z5\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649105 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-data\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649144 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-credential-socket\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istiod-ca-cert\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649288 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-certs\") pod \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\" (UID: \"548f5d51-cbf1-4553-aec3-cfc0de48fcae\") " Sep 11 11:48:32.649308 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649294 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:32.649565 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649546 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:32.649702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649686 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.649775 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.649708 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.650431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.650091 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:32.650431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.650396 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:48:32.650827 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.650795 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-data" (OuterVolumeSpecName: "istio-data") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:32.652076 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.652021 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:48:32.652189 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.652073 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-token" (OuterVolumeSpecName: "istio-token") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:48:32.652257 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.652227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-kube-api-access-d79z5" (OuterVolumeSpecName: "kube-api-access-d79z5") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "kube-api-access-d79z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:48:32.652630 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.652616 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "548f5d51-cbf1-4553-aec3-cfc0de48fcae" (UID: "548f5d51-cbf1-4553-aec3-cfc0de48fcae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 11:48:32.750596 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.750564 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.750596 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.750591 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.750596 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.750600 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.750818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.750608 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.750818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.750615 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.750818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.750622 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79z5\" (UniqueName: \"kubernetes.io/projected/548f5d51-cbf1-4553-aec3-cfc0de48fcae-kube-api-access-d79z5\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:32.750818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:32.750632 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548f5d51-cbf1-4553-aec3-cfc0de48fcae-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:48:33.613792 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:33.613759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g" Sep 11 11:48:33.647908 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:33.647878 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g"] Sep 11 11:48:33.654885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:33.654859 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bddd3ec1-openshift-default-868fcdcc56-rbg2g"] Sep 11 11:48:34.874844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:34.874811 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" path="/var/lib/kubelet/pods/548f5d51-cbf1-4553-aec3-cfc0de48fcae/volumes" Sep 11 11:48:43.690201 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:43.690160 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:48:47.640229 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:47.640191 2580 patch_prober.go:28] interesting pod/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 11:48:47.640654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:47.640258 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 11:48:48.723231 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:48.723198 2580 generic.go:356] "Generic (PLEG): container finished" podID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerID="bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51" exitCode=0 Sep 11 11:48:48.723653 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:48.723257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" event={"ID":"efe3f2a7-62b7-4da8-a0c4-895474b923e9","Type":"ContainerDied","Data":"bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51"} Sep 11 11:48:48.723653 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:48.723608 2580 scope.go:117] "RemoveContainer" containerID="bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51" Sep 11 11:48:48.724674 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:48.724644 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:48.724742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:48.724715 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:48.724778 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:48.724754 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:48:49.731371 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:49.731295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" event={"ID":"efe3f2a7-62b7-4da8-a0c4-895474b923e9","Type":"ContainerStarted","Data":"8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2"} Sep 11 11:48:50.611640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:50.611605 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:50.611894 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:50.611861 2580 patch_prober.go:28] interesting pod/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 11:48:50.612018 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:50.611931 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 11:48:51.612581 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:51.612541 2580 patch_prober.go:28] interesting pod/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 11:48:51.612972 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:51.612598 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 11:48:52.612075 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:52.612024 2580 patch_prober.go:28] interesting pod/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 11:48:52.612255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:52.612100 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 11:48:53.612670 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:53.612638 2580 patch_prober.go:28] interesting pod/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 11:48:53.613110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:53.612708 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 11:48:54.202232 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:54.202199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:48:54.612340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:54.612246 2580 patch_prober.go:28] interesting pod/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 11:48:54.612340 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:54.612305 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 11:48:55.615382 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:55.615345 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:55.615828 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:55.615642 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:55.616373 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:55.616358 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:48:58.861852 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:58.861820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:48:58.862352 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:48:58.862022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:49:03.128847 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:03.128808 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt"] Sep 11 11:49:03.129240 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:03.129083 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" containerID="cri-o://8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2" gracePeriod=30 Sep 11 11:49:08.385016 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.384992 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:49:08.417810 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.417781 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-token\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.417958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.417817 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-certs\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.417958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.417852 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-envoy\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.417958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.417883 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlm88\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-kube-api-access-zlm88\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.417958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.417924 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istiod-ca-cert\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.417958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.417947 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-data\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.418255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.417980 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-podinfo\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.418255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418073 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-socket\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.418255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418111 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:49:08.418255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-credential-socket\") pod \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\" (UID: \"efe3f2a7-62b7-4da8-a0c4-895474b923e9\") " Sep 11 11:49:08.418463 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418260 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:49:08.418463 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418344 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:49:08.418463 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418444 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:49:08.418647 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418393 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:49:08.418822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418797 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.418822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418823 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.418822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418836 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.418822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418849 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.418822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.418863 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.420342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.420312 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:49:08.420342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.420327 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 11:49:08.420461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.420380 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:49:08.420569 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.420552 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-kube-api-access-zlm88" (OuterVolumeSpecName: "kube-api-access-zlm88") pod "efe3f2a7-62b7-4da8-a0c4-895474b923e9" (UID: "efe3f2a7-62b7-4da8-a0c4-895474b923e9"). InnerVolumeSpecName "kube-api-access-zlm88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:49:08.520120 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.520083 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.520120 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.520111 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.520120 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.520120 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efe3f2a7-62b7-4da8-a0c4-895474b923e9-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.520348 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.520131 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlm88\" (UniqueName: \"kubernetes.io/projected/efe3f2a7-62b7-4da8-a0c4-895474b923e9-kube-api-access-zlm88\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:08.851254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.851173 2580 generic.go:356] "Generic (PLEG): container finished" podID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerID="8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2" exitCode=0 Sep 11 11:49:08.851254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.851220 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" Sep 11 11:49:08.851430 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.851221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" event={"ID":"efe3f2a7-62b7-4da8-a0c4-895474b923e9","Type":"ContainerDied","Data":"8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2"} Sep 11 11:49:08.851430 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.851362 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt" event={"ID":"efe3f2a7-62b7-4da8-a0c4-895474b923e9","Type":"ContainerDied","Data":"a6dd270cda2dfe64a1a567c1de7a656a233a4bee570081fc3f88b55fbe1df071"} Sep 11 11:49:08.851430 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.851398 2580 scope.go:117] "RemoveContainer" containerID="8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2" Sep 11 11:49:08.862195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.862176 2580 scope.go:117] "RemoveContainer" containerID="bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51" Sep 11 11:49:08.871323 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.871306 2580 scope.go:117] "RemoveContainer" containerID="8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2" Sep 11 11:49:08.871645 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:49:08.871623 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2\": container with ID starting with 8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2 not found: ID does not exist" containerID="8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2" Sep 11 11:49:08.871762 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.871656 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2"} err="failed to get container status \"8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2\": rpc error: code = NotFound desc = could not find container \"8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2\": container with ID starting with 8ab225a84c526e1cf8ae6262114d0f0a89afd4111eacbc2cdc05e182440b8fc2 not found: ID does not exist" Sep 11 11:49:08.871762 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.871748 2580 scope.go:117] "RemoveContainer" containerID="bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51" Sep 11 11:49:08.872150 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:49:08.872116 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51\": container with ID starting with bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51 not found: ID does not exist" containerID="bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51" Sep 11 11:49:08.872248 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.872157 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51"} err="failed to get container status \"bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51\": rpc error: code = NotFound desc = could not find container \"bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51\": container with ID starting with bf26295afc3a7c4d02f7d702d13d2b39ace745ddcbca171c0bb348416cd91c51 not found: ID does not exist" Sep 11 11:49:08.875214 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.875194 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt"] Sep 11 11:49:08.884100 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:08.884079 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5fdde1a2-gw-openshift-default-8458b596dd-xn4qt"] Sep 11 11:49:10.873158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:10.873129 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" path="/var/lib/kubelet/pods/efe3f2a7-62b7-4da8-a0c4-895474b923e9/volumes" Sep 11 11:49:14.403153 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:14.403116 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:49:32.104550 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:32.104507 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:49:34.755408 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755377 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd"] Sep 11 11:49:34.755779 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755766 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" Sep 11 11:49:34.755823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755781 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" Sep 11 11:49:34.755823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755800 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" Sep 11 11:49:34.755823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755805 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" Sep 11 11:49:34.755823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755816 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" Sep 11 11:49:34.755823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755822 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" Sep 11 11:49:34.755979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755893 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" Sep 11 11:49:34.755979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755900 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" Sep 11 11:49:34.755979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755907 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" Sep 11 11:49:34.755979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.755915 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe3f2a7-62b7-4da8-a0c4-895474b923e9" containerName="istio-proxy" Sep 11 11:49:34.758181 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.758166 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" Sep 11 11:49:34.760376 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.760352 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 11:49:34.760536 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.760379 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f46tj\"" Sep 11 11:49:34.767353 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.767328 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd"] Sep 11 11:49:34.880760 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.880730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0b767e-27cb-4542-9fb5-6fac292df8df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd\" (UID: \"6d0b767e-27cb-4542-9fb5-6fac292df8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" Sep 11 11:49:34.981912 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.981869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0b767e-27cb-4542-9fb5-6fac292df8df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd\" (UID: \"6d0b767e-27cb-4542-9fb5-6fac292df8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" Sep 11 11:49:34.982512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:34.982486 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0b767e-27cb-4542-9fb5-6fac292df8df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd\" (UID: \"6d0b767e-27cb-4542-9fb5-6fac292df8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" Sep 11 11:49:35.070202 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:35.070121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" Sep 11 11:49:35.342173 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:35.342145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd"] Sep 11 11:49:35.344005 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:49:35.343975 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0b767e_27cb_4542_9fb5_6fac292df8df.slice/crio-1ee21c497e162e3966b555b0fdf2fef30548c17192c1285b867526bffedaf16f WatchSource:0}: Error finding container 1ee21c497e162e3966b555b0fdf2fef30548c17192c1285b867526bffedaf16f: Status 404 returned error can't find the container with id 1ee21c497e162e3966b555b0fdf2fef30548c17192c1285b867526bffedaf16f Sep 11 11:49:36.008136 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:36.008102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" event={"ID":"6d0b767e-27cb-4542-9fb5-6fac292df8df","Type":"ContainerStarted","Data":"1ee21c497e162e3966b555b0fdf2fef30548c17192c1285b867526bffedaf16f"} Sep 11 11:49:38.025195 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:38.025158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" event={"ID":"6d0b767e-27cb-4542-9fb5-6fac292df8df","Type":"ContainerStarted","Data":"fc47214afe9b1a51f61df230d65c2ca70936f4950598f8107d39b496f21eb7da"} Sep 11 11:49:38.039838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:38.039743 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" podStartSLOduration=1.948814684 podStartE2EDuration="4.039728374s" totalImagesPullingTime="2.09091369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:49:38.039016918 +0000 UTC m=+939.673610055" watchObservedRunningTime="2026-09-11 11:49:38.039728374 +0000 UTC m=+939.674321504" Sep 11 11:49:39.032165 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:39.032135 2580 generic.go:356] "Generic (PLEG): container finished" podID="6d0b767e-27cb-4542-9fb5-6fac292df8df" containerID="fc47214afe9b1a51f61df230d65c2ca70936f4950598f8107d39b496f21eb7da" exitCode=0 Sep 11 11:49:39.032544 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:39.032225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" event={"ID":"6d0b767e-27cb-4542-9fb5-6fac292df8df","Type":"ContainerDied","Data":"fc47214afe9b1a51f61df230d65c2ca70936f4950598f8107d39b496f21eb7da"} Sep 11 11:49:40.176502 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:40.176478 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" Sep 11 11:49:40.337787 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:40.337698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0b767e-27cb-4542-9fb5-6fac292df8df-maas-api-service-ca\") pod \"6d0b767e-27cb-4542-9fb5-6fac292df8df\" (UID: \"6d0b767e-27cb-4542-9fb5-6fac292df8df\") " Sep 11 11:49:40.338090 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:40.338070 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0b767e-27cb-4542-9fb5-6fac292df8df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d0b767e-27cb-4542-9fb5-6fac292df8df" (UID: "6d0b767e-27cb-4542-9fb5-6fac292df8df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:49:40.439263 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:40.439231 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d0b767e-27cb-4542-9fb5-6fac292df8df-maas-api-service-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:49:41.051863 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:41.051838 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" Sep 11 11:49:41.052124 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:41.051826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd" event={"ID":"6d0b767e-27cb-4542-9fb5-6fac292df8df","Type":"ContainerDied","Data":"1ee21c497e162e3966b555b0fdf2fef30548c17192c1285b867526bffedaf16f"} Sep 11 11:49:41.052124 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:41.051987 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ee21c497e162e3966b555b0fdf2fef30548c17192c1285b867526bffedaf16f" Sep 11 11:49:45.942903 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:45.942863 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd"] Sep 11 11:49:45.947216 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:45.947181 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5ab7-q7pbd"] Sep 11 11:49:46.873963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:46.873932 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d0b767e-27cb-4542-9fb5-6fac292df8df" path="/var/lib/kubelet/pods/6d0b767e-27cb-4542-9fb5-6fac292df8df/volumes" Sep 11 11:49:50.305805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:49:50.305768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:50:07.330951 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.330916 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl"] Sep 11 11:50:07.331356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.331330 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d0b767e-27cb-4542-9fb5-6fac292df8df" containerName="revoke-keys" Sep 11 11:50:07.331356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.331342 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d0b767e-27cb-4542-9fb5-6fac292df8df" containerName="revoke-keys" Sep 11 11:50:07.331356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.331352 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" Sep 11 11:50:07.331356 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.331357 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="548f5d51-cbf1-4553-aec3-cfc0de48fcae" containerName="istio-proxy" Sep 11 11:50:07.331491 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.331440 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d0b767e-27cb-4542-9fb5-6fac292df8df" containerName="revoke-keys" Sep 11 11:50:07.337380 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.337359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.340332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.340311 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dbe0e7\"/\"openshift-service-ca.crt\"" Sep 11 11:50:07.340332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.340322 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dbe0e7\"/\"default-dockercfg-lkm5j\"" Sep 11 11:50:07.340509 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.340369 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dbe0e7\"/\"test-model-dbe0e7-kserve-self-signed-certs\"" Sep 11 11:50:07.340509 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.340414 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dbe0e7\"/\"kube-root-ca.crt\"" Sep 11 11:50:07.347505 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.347481 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl"] Sep 11 11:50:07.405641 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.405600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj9pf\" (UniqueName: \"kubernetes.io/projected/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-kube-api-access-bj9pf\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.405822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.405672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tmp-dir\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.405822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.405706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-dshm\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.405822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.405736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-model-cache\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.405822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.405767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tls-certs\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.405822 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.405801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-home\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.506469 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.506428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tls-certs\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.506666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.506483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-home\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.506666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.506516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj9pf\" (UniqueName: \"kubernetes.io/projected/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-kube-api-access-bj9pf\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.506666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.506549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tmp-dir\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.506666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.506574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-dshm\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.506666 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.506610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-model-cache\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.507016 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.506986 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-home\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.507114 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.507025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-model-cache\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.507114 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.507082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tmp-dir\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.509155 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.509124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-dshm\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.509321 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.509299 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tls-certs\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.516154 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.516124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj9pf\" (UniqueName: \"kubernetes.io/projected/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-kube-api-access-bj9pf\") pod \"test-model-dbe0e7-kserve-6cbf7498d7-52gcl\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.646774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.646684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:07.931503 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:07.931471 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl"] Sep 11 11:50:07.933390 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:50:07.933365 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e292a45_be1b_4c90_83a8_ff5ebe5bb6e4.slice/crio-98b0a2f4e22ad7c9a9151c5a37ce3e8a0f7e7bf12c48fbbce7fdf1d14ab08a66 WatchSource:0}: Error finding container 98b0a2f4e22ad7c9a9151c5a37ce3e8a0f7e7bf12c48fbbce7fdf1d14ab08a66: Status 404 returned error can't find the container with id 98b0a2f4e22ad7c9a9151c5a37ce3e8a0f7e7bf12c48fbbce7fdf1d14ab08a66 Sep 11 11:50:08.222614 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:08.222524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" event={"ID":"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4","Type":"ContainerStarted","Data":"3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211"} Sep 11 11:50:08.222614 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:08.222574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" event={"ID":"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4","Type":"ContainerStarted","Data":"98b0a2f4e22ad7c9a9151c5a37ce3e8a0f7e7bf12c48fbbce7fdf1d14ab08a66"} Sep 11 11:50:08.241417 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:08.241363 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" podStartSLOduration=1.241340848 podStartE2EDuration="1.241340848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:50:08.238752513 +0000 UTC m=+969.873345646" watchObservedRunningTime="2026-09-11 11:50:08.241340848 +0000 UTC m=+969.875933981" Sep 11 11:50:11.494671 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:11.494637 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:50:14.316861 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:14.316828 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6"] Sep 11 11:50:14.317317 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:14.317089 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" containerID="cri-o://48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82" gracePeriod=30 Sep 11 11:50:14.959878 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:14.959836 2580 patch_prober.go:28] interesting pod/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 11:50:14.960068 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:14.959909 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 11:50:17.647345 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:17.647310 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:17.647345 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:17.647353 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:17.659724 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:17.659700 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:18.298179 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:18.298152 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:50:19.570389 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.570366 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:50:19.631504 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631427 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-certs\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.631504 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631467 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istiod-ca-cert\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.631730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631538 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8krj\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-kube-api-access-j8krj\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.631730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631568 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-data\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.631730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-socket\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.631730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631650 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-envoy\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.631730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631673 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-token\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.631730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631713 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-credential-socket\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.632030 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631744 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-podinfo\") pod \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\" (UID: \"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d\") " Sep 11 11:50:19.632134 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631740 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:50:19.632134 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631913 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:50:19.632134 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631925 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:50:19.632134 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.631964 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:50:19.632134 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.632118 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:50:19.633963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.633941 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:50:19.634097 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.633996 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 11:50:19.634317 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.634300 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-kube-api-access-j8krj" (OuterVolumeSpecName: "kube-api-access-j8krj") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "kube-api-access-j8krj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:50:19.634317 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.634307 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" (UID: "1d4ae0e2-119c-4f79-aedf-54c9f1ac471d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:50:19.733110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733024 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733072 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733082 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733089 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733098 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733106 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733110 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733114 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733746 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733122 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8krj\" (UniqueName: \"kubernetes.io/projected/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-kube-api-access-j8krj\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:19.733746 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:19.733130 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:50:20.300398 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.300368 2580 generic.go:356] "Generic (PLEG): container finished" podID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerID="48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82" exitCode=0 Sep 11 11:50:20.300582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.300432 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" Sep 11 11:50:20.300582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.300448 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" event={"ID":"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d","Type":"ContainerDied","Data":"48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82"} Sep 11 11:50:20.300582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.300503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6" event={"ID":"1d4ae0e2-119c-4f79-aedf-54c9f1ac471d","Type":"ContainerDied","Data":"1e99b5f75bf1d76b1cb78888069e49fb8a270bcd340ee9a44b77a1e62d23620e"} Sep 11 11:50:20.300582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.300535 2580 scope.go:117] "RemoveContainer" containerID="48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82" Sep 11 11:50:20.311149 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.311133 2580 scope.go:117] "RemoveContainer" containerID="de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0" Sep 11 11:50:20.320207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.320186 2580 scope.go:117] "RemoveContainer" containerID="48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82" Sep 11 11:50:20.320530 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:50:20.320487 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82\": container with ID starting with 48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82 not found: ID does not exist" containerID="48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82" Sep 11 11:50:20.320621 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.320527 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82"} err="failed to get container status \"48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82\": rpc error: code = NotFound desc = could not find container \"48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82\": container with ID starting with 48ab6ef98eb8588862e39e32788da4c0fdbd455281ea33be338c6ff1ff871b82 not found: ID does not exist" Sep 11 11:50:20.320621 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.320546 2580 scope.go:117] "RemoveContainer" containerID="de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0" Sep 11 11:50:20.320839 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:50:20.320823 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0\": container with ID starting with de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0 not found: ID does not exist" containerID="de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0" Sep 11 11:50:20.320882 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.320844 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0"} err="failed to get container status \"de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0\": rpc error: code = NotFound desc = could not find container \"de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0\": container with ID starting with de1996639e4447e62157dc92ca4e3aa788ff8a6259f6a2ad4631143f719c23c0 not found: ID does not exist" Sep 11 11:50:20.322477 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.322460 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6"] Sep 11 11:50:20.328104 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.328084 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5a5ab7-openshift-default-8664dcff6d-dcwm6"] Sep 11 11:50:20.873771 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:20.873741 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" path="/var/lib/kubelet/pods/1d4ae0e2-119c-4f79-aedf-54c9f1ac471d/volumes" Sep 11 11:50:34.414321 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:34.414280 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:50:44.852549 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:44.852505 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2"] Sep 11 11:50:50.396431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:50.396396 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:50:59.875343 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:50:59.875301 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" podUID="7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" containerName="main" containerID="cri-o://b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f" gracePeriod=45 Sep 11 11:51:00.130111 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.130027 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:51:00.216367 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216330 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tls-certs\") pod \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " Sep 11 11:51:00.216582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216409 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkjrt\" (UniqueName: \"kubernetes.io/projected/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-kube-api-access-fkjrt\") pod \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " Sep 11 11:51:00.216582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216461 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-home\") pod \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " Sep 11 11:51:00.216582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216501 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-dshm\") pod \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " Sep 11 11:51:00.216582 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216562 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tmp-dir\") pod \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " Sep 11 11:51:00.216798 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216591 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-model-cache\") pod \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\" (UID: \"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5\") " Sep 11 11:51:00.216898 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216866 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" (UID: "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:51:00.216963 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216876 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-home" (OuterVolumeSpecName: "home") pod "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" (UID: "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:51:00.217056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.216988 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" (UID: "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:51:00.218788 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.218768 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-dshm" (OuterVolumeSpecName: "dshm") pod "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" (UID: "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:51:00.218887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.218819 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" (UID: "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:51:00.218887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.218881 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-kube-api-access-fkjrt" (OuterVolumeSpecName: "kube-api-access-fkjrt") pod "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" (UID: "7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5"). InnerVolumeSpecName "kube-api-access-fkjrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:51:00.318432 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.318397 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.318432 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.318427 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.318432 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.318435 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.318647 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.318444 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkjrt\" (UniqueName: \"kubernetes.io/projected/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-kube-api-access-fkjrt\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.318647 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.318453 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.318647 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.318460 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:51:00.543211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.543177 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" containerID="b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f" exitCode=0 Sep 11 11:51:00.543351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.543225 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" Sep 11 11:51:00.543351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.543250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" event={"ID":"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5","Type":"ContainerDied","Data":"b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f"} Sep 11 11:51:00.543351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.543282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2" event={"ID":"7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5","Type":"ContainerDied","Data":"ea3d22b698e89375065a824f7aa3525f48a3a64987491b77dab24afe832ec559"} Sep 11 11:51:00.543351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.543298 2580 scope.go:117] "RemoveContainer" containerID="b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f" Sep 11 11:51:00.552908 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.552888 2580 scope.go:117] "RemoveContainer" containerID="b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f" Sep 11 11:51:00.553212 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:51:00.553193 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f\": container with ID starting with b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f not found: ID does not exist" containerID="b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f" Sep 11 11:51:00.553266 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.553220 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f"} err="failed to get container status \"b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f\": rpc error: code = NotFound desc = could not find container \"b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f\": container with ID starting with b26fbe68415c3081aefecbec71f184785251b304839b9671880ada2f6935e72f not found: ID does not exist" Sep 11 11:51:00.566102 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.566077 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2"] Sep 11 11:51:00.572880 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.572860 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-31a432b7-kserve-6c765b888d-nkwg2"] Sep 11 11:51:00.873288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:00.873216 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" path="/var/lib/kubelet/pods/7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5/volumes" Sep 11 11:51:03.996949 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:03.996918 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:51:11.822652 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.822617 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj"] Sep 11 11:51:11.823301 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823280 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" Sep 11 11:51:11.823350 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823306 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" Sep 11 11:51:11.823350 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823328 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" Sep 11 11:51:11.823350 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823336 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" Sep 11 11:51:11.823451 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823362 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" containerName="main" Sep 11 11:51:11.823451 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823371 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" containerName="main" Sep 11 11:51:11.823530 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823518 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" Sep 11 11:51:11.823565 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823538 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4ae0e2-119c-4f79-aedf-54c9f1ac471d" containerName="istio-proxy" Sep 11 11:51:11.823565 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.823549 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f4d4e75-0326-497e-92c8-5f9fd4e4c5f5" containerName="main" Sep 11 11:51:11.828591 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.828570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:11.833254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.833222 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6bcf5\"/\"kube-root-ca.crt\"" Sep 11 11:51:11.833777 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.833754 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6bcf5\"/\"test-model-c6bcf5-kserve-self-signed-certs\"" Sep 11 11:51:11.834499 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.834138 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c6bcf5\"/\"default-dockercfg-gh9z5\"" Sep 11 11:51:11.834499 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.834413 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c6bcf5\"/\"openshift-service-ca.crt\"" Sep 11 11:51:11.838201 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.838112 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj"] Sep 11 11:51:11.927902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.927872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkds\" (UniqueName: \"kubernetes.io/projected/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-kube-api-access-zpkds\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:11.928066 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.927910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tls-certs\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:11.928066 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.927936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-dshm\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:11.928158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.928065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tmp-dir\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:11.928158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.928097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-model-cache\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:11.928158 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:11.928117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-home\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.029678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.029641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-dshm\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.029871 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.029727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tmp-dir\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.029871 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.029753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-model-cache\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.029871 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.029778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-home\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.029871 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.029834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkds\" (UniqueName: \"kubernetes.io/projected/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-kube-api-access-zpkds\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.029871 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.029865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tls-certs\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.030183 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.030156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tmp-dir\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.030246 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.030213 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-model-cache\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.030246 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.030237 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-home\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.032021 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.032002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-dshm\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.032366 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.032349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tls-certs\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.038088 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.038068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkds\" (UniqueName: \"kubernetes.io/projected/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-kube-api-access-zpkds\") pod \"test-model-c6bcf5-kserve-6f775f5979-sjjbj\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.144810 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.144734 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:12.420823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.420790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj"] Sep 11 11:51:12.421952 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:51:12.421928 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ae1d2df_19d8_40dc_8180_bc030f72d1e9.slice/crio-e0066420551e073ed5bece64964acd87fb845686b6196e60147c703a6dd813a8 WatchSource:0}: Error finding container e0066420551e073ed5bece64964acd87fb845686b6196e60147c703a6dd813a8: Status 404 returned error can't find the container with id e0066420551e073ed5bece64964acd87fb845686b6196e60147c703a6dd813a8 Sep 11 11:51:12.615685 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.615646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" event={"ID":"6ae1d2df-19d8-40dc-8180-bc030f72d1e9","Type":"ContainerStarted","Data":"bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03"} Sep 11 11:51:12.615685 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.615695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" event={"ID":"6ae1d2df-19d8-40dc-8180-bc030f72d1e9","Type":"ContainerStarted","Data":"e0066420551e073ed5bece64964acd87fb845686b6196e60147c703a6dd813a8"} Sep 11 11:51:12.631784 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:12.631738 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" podStartSLOduration=1.6317223109999999 podStartE2EDuration="1.631722311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:51:12.631145794 +0000 UTC m=+1034.265738939" watchObservedRunningTime="2026-09-11 11:51:12.631722311 +0000 UTC m=+1034.266315444" Sep 11 11:51:21.017197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:21.017162 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:51:22.145706 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:22.145671 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:22.145706 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:22.145715 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:22.158103 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:22.158080 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:22.690155 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:22.690124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:51:37.203007 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:37.202975 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:51:54.408976 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:51:54.408888 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:52:14.206051 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:14.205994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:52:23.275711 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:23.275679 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl"] Sep 11 11:52:24.295013 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:24.294970 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj"] Sep 11 11:52:27.698120 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:27.698083 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:52:38.264529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.264488 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:52:38.295530 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.295489 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" podUID="6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" containerName="main" containerID="cri-o://3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211" gracePeriod=45 Sep 11 11:52:38.299481 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.299439 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" podUID="6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": read tcp 10.132.0.2:58850->10.132.0.57:8000: read: connection reset by peer" Sep 11 11:52:38.549531 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.549508 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:52:38.651461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651430 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-home\") pod \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " Sep 11 11:52:38.651461 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651466 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-dshm\") pod \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " Sep 11 11:52:38.651660 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651507 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj9pf\" (UniqueName: \"kubernetes.io/projected/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-kube-api-access-bj9pf\") pod \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " Sep 11 11:52:38.651660 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651559 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tmp-dir\") pod \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " Sep 11 11:52:38.651757 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651700 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tls-certs\") pod \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " Sep 11 11:52:38.651757 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-home" (OuterVolumeSpecName: "home") pod "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" (UID: "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:38.651863 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651785 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-model-cache\") pod \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\" (UID: \"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4\") " Sep 11 11:52:38.651863 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.651796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" (UID: "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:38.652058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.652015 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" (UID: "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:38.652205 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.652187 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:38.652265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.652209 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:38.652265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.652218 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:38.653966 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.653935 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" (UID: "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:52:38.653966 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.653957 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-dshm" (OuterVolumeSpecName: "dshm") pod "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" (UID: "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:38.654128 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.653968 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-kube-api-access-bj9pf" (OuterVolumeSpecName: "kube-api-access-bj9pf") pod "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" (UID: "6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4"). InnerVolumeSpecName "kube-api-access-bj9pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:52:38.752930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.752897 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:38.752930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.752924 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj9pf\" (UniqueName: \"kubernetes.io/projected/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-kube-api-access-bj9pf\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:38.752930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:38.752935 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:39.112766 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.112734 2580 generic.go:356] "Generic (PLEG): container finished" podID="6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" containerID="3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211" exitCode=0 Sep 11 11:52:39.112928 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.112776 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" Sep 11 11:52:39.112928 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.112822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" event={"ID":"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4","Type":"ContainerDied","Data":"3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211"} Sep 11 11:52:39.112928 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.112867 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl" event={"ID":"6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4","Type":"ContainerDied","Data":"98b0a2f4e22ad7c9a9151c5a37ce3e8a0f7e7bf12c48fbbce7fdf1d14ab08a66"} Sep 11 11:52:39.112928 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.112885 2580 scope.go:117] "RemoveContainer" containerID="3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211" Sep 11 11:52:39.124742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.124722 2580 scope.go:117] "RemoveContainer" containerID="3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211" Sep 11 11:52:39.125021 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:52:39.125003 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211\": container with ID starting with 3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211 not found: ID does not exist" containerID="3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211" Sep 11 11:52:39.125113 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.125051 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211"} err="failed to get container status \"3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211\": rpc error: code = NotFound desc = could not find container \"3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211\": container with ID starting with 3ca5552095e1c9ba68793ec1d62b6a020ef05993709fed9f1ce0a5e11f58a211 not found: ID does not exist" Sep 11 11:52:39.145941 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.145881 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl"] Sep 11 11:52:39.152271 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.152249 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dbe0e7/test-model-dbe0e7-kserve-6cbf7498d7-52gcl"] Sep 11 11:52:39.314521 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.314477 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" podUID="6ae1d2df-19d8-40dc-8180-bc030f72d1e9" containerName="main" containerID="cri-o://bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03" gracePeriod=45 Sep 11 11:52:39.569274 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.569253 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:52:39.660942 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.660916 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpkds\" (UniqueName: \"kubernetes.io/projected/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-kube-api-access-zpkds\") pod \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " Sep 11 11:52:39.661150 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.660951 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-dshm\") pod \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " Sep 11 11:52:39.661150 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.660986 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tls-certs\") pod \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " Sep 11 11:52:39.661150 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661006 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-home\") pod \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " Sep 11 11:52:39.661150 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661071 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tmp-dir\") pod \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " Sep 11 11:52:39.661150 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661103 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-model-cache\") pod \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\" (UID: \"6ae1d2df-19d8-40dc-8180-bc030f72d1e9\") " Sep 11 11:52:39.661437 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661339 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-home" (OuterVolumeSpecName: "home") pod "6ae1d2df-19d8-40dc-8180-bc030f72d1e9" (UID: "6ae1d2df-19d8-40dc-8180-bc030f72d1e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:39.661437 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661348 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ae1d2df-19d8-40dc-8180-bc030f72d1e9" (UID: "6ae1d2df-19d8-40dc-8180-bc030f72d1e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:39.661527 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661443 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:39.661527 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661452 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ae1d2df-19d8-40dc-8180-bc030f72d1e9" (UID: "6ae1d2df-19d8-40dc-8180-bc030f72d1e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:39.661527 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.661462 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:39.663476 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.663448 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-dshm" (OuterVolumeSpecName: "dshm") pod "6ae1d2df-19d8-40dc-8180-bc030f72d1e9" (UID: "6ae1d2df-19d8-40dc-8180-bc030f72d1e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:39.663578 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.663506 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ae1d2df-19d8-40dc-8180-bc030f72d1e9" (UID: "6ae1d2df-19d8-40dc-8180-bc030f72d1e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:52:39.663578 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.663522 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-kube-api-access-zpkds" (OuterVolumeSpecName: "kube-api-access-zpkds") pod "6ae1d2df-19d8-40dc-8180-bc030f72d1e9" (UID: "6ae1d2df-19d8-40dc-8180-bc030f72d1e9"). InnerVolumeSpecName "kube-api-access-zpkds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:52:39.762501 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.762421 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkds\" (UniqueName: \"kubernetes.io/projected/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-kube-api-access-zpkds\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:39.762501 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.762446 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:39.762501 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.762456 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:39.762501 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:39.762464 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae1d2df-19d8-40dc-8180-bc030f72d1e9-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:40.120117 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.119997 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ae1d2df-19d8-40dc-8180-bc030f72d1e9" containerID="bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03" exitCode=0 Sep 11 11:52:40.120117 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.120079 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" Sep 11 11:52:40.120117 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.120081 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" event={"ID":"6ae1d2df-19d8-40dc-8180-bc030f72d1e9","Type":"ContainerDied","Data":"bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03"} Sep 11 11:52:40.120404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.120124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj" event={"ID":"6ae1d2df-19d8-40dc-8180-bc030f72d1e9","Type":"ContainerDied","Data":"e0066420551e073ed5bece64964acd87fb845686b6196e60147c703a6dd813a8"} Sep 11 11:52:40.120404 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.120147 2580 scope.go:117] "RemoveContainer" containerID="bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03" Sep 11 11:52:40.131268 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.131246 2580 scope.go:117] "RemoveContainer" containerID="bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03" Sep 11 11:52:40.131567 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:52:40.131546 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03\": container with ID starting with bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03 not found: ID does not exist" containerID="bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03" Sep 11 11:52:40.131648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.131579 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03"} err="failed to get container status \"bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03\": rpc error: code = NotFound desc = could not find container \"bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03\": container with ID starting with bdc9866890d5abfb7277f216f72e0499b9a99c2542838feaa776073ead3b8f03 not found: ID does not exist" Sep 11 11:52:40.155577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.155554 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj"] Sep 11 11:52:40.162080 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.162058 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c6bcf5/test-model-c6bcf5-kserve-6f775f5979-sjjbj"] Sep 11 11:52:40.873341 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.873307 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ae1d2df-19d8-40dc-8180-bc030f72d1e9" path="/var/lib/kubelet/pods/6ae1d2df-19d8-40dc-8180-bc030f72d1e9/volumes" Sep 11 11:52:40.873876 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:40.873853 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" path="/var/lib/kubelet/pods/6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4/volumes" Sep 11 11:52:43.676968 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:43.676938 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q"] Sep 11 11:52:43.677479 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:43.677173 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" containerID="cri-o://d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2" gracePeriod=30 Sep 11 11:52:44.584400 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:44.584367 2580 patch_prober.go:28] interesting pod/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 11:52:44.584580 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:44.584424 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 11:52:48.023697 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:48.023658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:52:48.932955 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:48.932927 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:52:49.054232 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054135 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-podinfo\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054232 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054190 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-token\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054232 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054218 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-data\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054330 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljwrg\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-kube-api-access-ljwrg\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-certs\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054456 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-envoy\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054478 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054498 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-credential-socket\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054544 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istiod-ca-cert\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054576 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-socket\") pod \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\" (UID: \"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee\") " Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054692 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:49.054818 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054773 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:49.055486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.054998 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.055486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.055086 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:52:49.055486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.055167 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.055486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.055191 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:49.055486 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.055203 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.056701 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.056674 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:52:49.056701 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.056674 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 11:52:49.056883 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.056677 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:52:49.056883 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.056802 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-kube-api-access-ljwrg" (OuterVolumeSpecName: "kube-api-access-ljwrg") pod "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" (UID: "b7c8adfc-0d16-4e57-be2c-2d0f2d690eee"). InnerVolumeSpecName "kube-api-access-ljwrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:52:49.156386 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.156350 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.156386 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.156379 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.156386 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.156389 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.156386 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.156397 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.156654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.156406 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljwrg\" (UniqueName: \"kubernetes.io/projected/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-kube-api-access-ljwrg\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.156654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.156415 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:52:49.177202 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.177170 2580 generic.go:356] "Generic (PLEG): container finished" podID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerID="d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2" exitCode=0 Sep 11 11:52:49.177389 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.177236 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" Sep 11 11:52:49.177389 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.177244 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" event={"ID":"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee","Type":"ContainerDied","Data":"d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2"} Sep 11 11:52:49.177389 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.177310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q" event={"ID":"b7c8adfc-0d16-4e57-be2c-2d0f2d690eee","Type":"ContainerDied","Data":"0b542565d6dd8e5d132d898d137bba9ed736c70e0aebd6c7a2e942e501dc4757"} Sep 11 11:52:49.177389 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.177345 2580 scope.go:117] "RemoveContainer" containerID="d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2" Sep 11 11:52:49.187127 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.187111 2580 scope.go:117] "RemoveContainer" containerID="dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505" Sep 11 11:52:49.195978 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.195960 2580 scope.go:117] "RemoveContainer" containerID="d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2" Sep 11 11:52:49.196305 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:52:49.196288 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2\": container with ID starting with d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2 not found: ID does not exist" containerID="d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2" Sep 11 11:52:49.196360 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.196312 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2"} err="failed to get container status \"d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2\": rpc error: code = NotFound desc = could not find container \"d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2\": container with ID starting with d5b0cdec07e13b130293716b972a9933e08c169e165615e25547c60c0524b0d2 not found: ID does not exist" Sep 11 11:52:49.196360 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.196328 2580 scope.go:117] "RemoveContainer" containerID="dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505" Sep 11 11:52:49.196577 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:52:49.196562 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505\": container with ID starting with dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505 not found: ID does not exist" containerID="dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505" Sep 11 11:52:49.196620 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.196582 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505"} err="failed to get container status \"dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505\": rpc error: code = NotFound desc = could not find container \"dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505\": container with ID starting with dcec12d2b1af4bde2771984a34996a58f6bf9c25bbfb8d9b8bf3ec3cc3d8e505 not found: ID does not exist" Sep 11 11:52:49.203476 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.203455 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q"] Sep 11 11:52:49.207075 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:49.207057 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dbe0e7-openshift-default-694b6fb884-zr98q"] Sep 11 11:52:50.873109 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:50.873075 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" path="/var/lib/kubelet/pods/b7c8adfc-0d16-4e57-be2c-2d0f2d690eee/volumes" Sep 11 11:52:57.564720 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:52:57.564685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:53:02.969091 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:02.969061 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw"] Sep 11 11:53:02.969527 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:02.969287 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" containerID="cri-o://d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6" gracePeriod=30 Sep 11 11:53:08.224085 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.224056 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:53:08.288426 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.288340 2580 generic.go:356] "Generic (PLEG): container finished" podID="4133241f-99b7-481f-9977-925126d8407b" containerID="d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6" exitCode=0 Sep 11 11:53:08.288426 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.288385 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" Sep 11 11:53:08.288634 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.288445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" event={"ID":"4133241f-99b7-481f-9977-925126d8407b","Type":"ContainerDied","Data":"d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6"} Sep 11 11:53:08.288634 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.288523 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw" event={"ID":"4133241f-99b7-481f-9977-925126d8407b","Type":"ContainerDied","Data":"b6959762a4296f60ae8102b83c368b9d7d30854ab7f4662b6e59dd78a3579575"} Sep 11 11:53:08.288634 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.288554 2580 scope.go:117] "RemoveContainer" containerID="d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6" Sep 11 11:53:08.297707 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.297690 2580 scope.go:117] "RemoveContainer" containerID="d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a" Sep 11 11:53:08.305925 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.305907 2580 scope.go:117] "RemoveContainer" containerID="d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6" Sep 11 11:53:08.306209 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:53:08.306186 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6\": container with ID starting with d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6 not found: ID does not exist" containerID="d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6" Sep 11 11:53:08.306266 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.306219 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6"} err="failed to get container status \"d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6\": rpc error: code = NotFound desc = could not find container \"d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6\": container with ID starting with d2b4e6706533163423ea7655a03a024b43deb19040733c52e55e3139ce7491f6 not found: ID does not exist" Sep 11 11:53:08.306266 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.306238 2580 scope.go:117] "RemoveContainer" containerID="d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a" Sep 11 11:53:08.306472 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:53:08.306457 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a\": container with ID starting with d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a not found: ID does not exist" containerID="d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a" Sep 11 11:53:08.306529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.306475 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a"} err="failed to get container status \"d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a\": rpc error: code = NotFound desc = could not find container \"d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a\": container with ID starting with d93b554ab50bab9ba5cb6437980015bf2c72873405411c4341300bb789be979a not found: ID does not exist" Sep 11 11:53:08.329837 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.329815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4133241f-99b7-481f-9977-925126d8407b-istio-podinfo\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.329933 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.329872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-credential-socket\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.329933 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.329897 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2852r\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-kube-api-access-2852r\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.330053 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.329947 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-envoy\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.330053 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.329969 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-socket\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.330053 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.329986 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-data\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.330053 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330021 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-istio-token\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.330251 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330110 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-certs\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.330251 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330136 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4133241f-99b7-481f-9977-925126d8407b-istiod-ca-cert\") pod \"4133241f-99b7-481f-9977-925126d8407b\" (UID: \"4133241f-99b7-481f-9977-925126d8407b\") " Sep 11 11:53:08.330359 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330256 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:53:08.330359 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:53:08.330480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-data" (OuterVolumeSpecName: "istio-data") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:53:08.330635 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330606 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.330635 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330626 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4133241f-99b7-481f-9977-925126d8407b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:53:08.330635 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330633 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.330635 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330655 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.331058 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.330635 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:53:08.332139 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.332119 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4133241f-99b7-481f-9977-925126d8407b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 11:53:08.332326 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.332305 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-kube-api-access-2852r" (OuterVolumeSpecName: "kube-api-access-2852r") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "kube-api-access-2852r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:53:08.332421 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.332340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-istio-token" (OuterVolumeSpecName: "istio-token") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:53:08.332509 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.332491 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4133241f-99b7-481f-9977-925126d8407b" (UID: "4133241f-99b7-481f-9977-925126d8407b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:53:08.432054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.432007 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.432054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.432057 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4133241f-99b7-481f-9977-925126d8407b-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.432249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.432066 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4133241f-99b7-481f-9977-925126d8407b-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.432249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.432074 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2852r\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-kube-api-access-2852r\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.432249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.432084 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4133241f-99b7-481f-9977-925126d8407b-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.432249 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.432092 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4133241f-99b7-481f-9977-925126d8407b-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:53:08.624103 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.624075 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw"] Sep 11 11:53:08.643740 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.643717 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c6bcf5-openshift-default-577d9b8cfd-x27jw"] Sep 11 11:53:08.874203 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:08.874117 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4133241f-99b7-481f-9977-925126d8407b" path="/var/lib/kubelet/pods/4133241f-99b7-481f-9977-925126d8407b/volumes" Sep 11 11:53:11.879729 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:11.879695 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:53:25.114244 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:25.114166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:53:32.590550 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590519 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f"] Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590880 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590890 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590903 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590908 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590915 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590920 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590926 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ae1d2df-19d8-40dc-8180-bc030f72d1e9" containerName="main" Sep 11 11:53:32.590930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590931 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ae1d2df-19d8-40dc-8180-bc030f72d1e9" containerName="main" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590942 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590947 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590955 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" containerName="main" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.590960 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" containerName="main" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.591015 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e292a45-be1b-4c90-83a8-ff5ebe5bb6e4" containerName="main" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.591023 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.591029 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ae1d2df-19d8-40dc-8180-bc030f72d1e9" containerName="main" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.591049 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" Sep 11 11:53:32.591197 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.591057 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c8adfc-0d16-4e57-be2c-2d0f2d690eee" containerName="istio-proxy" Sep 11 11:53:32.594238 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.594221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.601431 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.601410 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-117711-openshift-default-dockercfg-c5w28\"" Sep 11 11:53:32.636553 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.636515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f"] Sep 11 11:53:32.651183 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-socket\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-certs\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-data\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-envoy\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-token\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651351 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqml5\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-kube-api-access-lqml5\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651646 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-podinfo\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651646 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-credential-socket\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.651646 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.651416 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1a741-2564-486d-910c-e2c33bf91e94-istiod-ca-cert\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752021 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.751987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-envoy\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752021 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-token\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqml5\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-kube-api-access-lqml5\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-podinfo\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-credential-socket\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1a741-2564-486d-910c-e2c33bf91e94-istiod-ca-cert\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-socket\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752288 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-certs\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-data\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-credential-socket\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-socket\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.752691 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-data\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.753100 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.752760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-certs\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.753265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.753238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1a741-2564-486d-910c-e2c33bf91e94-istiod-ca-cert\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.754627 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.754602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-envoy\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.754772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.754753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-podinfo\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.759802 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.759782 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqml5\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-kube-api-access-lqml5\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.759975 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.759959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-token\") pod \"e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:32.903845 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:32.903815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:33.187947 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.187914 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f"] Sep 11 11:53:33.189190 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:53:33.189161 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f1a741_2564_486d_910c_e2c33bf91e94.slice/crio-2f7f56e1ec2bac9a0992af9d0bb54ff8cb166359b43534dacee49b908ba5be0c WatchSource:0}: Error finding container 2f7f56e1ec2bac9a0992af9d0bb54ff8cb166359b43534dacee49b908ba5be0c: Status 404 returned error can't find the container with id 2f7f56e1ec2bac9a0992af9d0bb54ff8cb166359b43534dacee49b908ba5be0c Sep 11 11:53:33.191627 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.191584 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:33.191712 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.191676 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:33.191751 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.191716 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:33.433779 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.433742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" event={"ID":"e3f1a741-2564-486d-910c-e2c33bf91e94","Type":"ContainerStarted","Data":"1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748"} Sep 11 11:53:33.433942 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.433806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" event={"ID":"e3f1a741-2564-486d-910c-e2c33bf91e94","Type":"ContainerStarted","Data":"2f7f56e1ec2bac9a0992af9d0bb54ff8cb166359b43534dacee49b908ba5be0c"} Sep 11 11:53:33.461194 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.461090 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podStartSLOduration=1.46106822 podStartE2EDuration="1.46106822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:53:33.458795097 +0000 UTC m=+1175.093388229" watchObservedRunningTime="2026-09-11 11:53:33.46106822 +0000 UTC m=+1175.095661360" Sep 11 11:53:33.904010 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:33.903981 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:34.910117 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:34.910085 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:35.445772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:35.445738 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:35.446705 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:35.446682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:37.311132 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.311098 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6"] Sep 11 11:53:37.311556 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.311543 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4133241f-99b7-481f-9977-925126d8407b" containerName="istio-proxy" Sep 11 11:53:37.314994 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.314977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.320741 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.320694 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 11:53:37.320741 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.320730 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-117711-serving-cert\"" Sep 11 11:53:37.320951 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.320871 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 11:53:37.321648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.321619 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tztb5\"" Sep 11 11:53:37.323424 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.323405 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 11:53:37.336685 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.336659 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6"] Sep 11 11:53:37.396576 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.396525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5dmg\" (UniqueName: \"kubernetes.io/projected/2a8c8284-f727-43ae-854e-c60c03b0438b-kube-api-access-v5dmg\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.396760 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.396590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.396760 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.396633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.396760 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.396663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c8284-f727-43ae-854e-c60c03b0438b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.498086 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.498023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5dmg\" (UniqueName: \"kubernetes.io/projected/2a8c8284-f727-43ae-854e-c60c03b0438b-kube-api-access-v5dmg\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.498285 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.498113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.498285 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.498168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.498285 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.498199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c8284-f727-43ae-854e-c60c03b0438b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.498285 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:53:37.498261 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-serving-cert: secret "maas-api-e2e-shared-b-w3-117711-serving-cert" not found Sep 11 11:53:37.498525 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:53:37.498320 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls podName:2a8c8284-f727-43ae-854e-c60c03b0438b nodeName:}" failed. No retries permitted until 2026-09-11 11:53:37.998305033 +0000 UTC m=+1179.632898143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" (UID: "2a8c8284-f727-43ae-854e-c60c03b0438b") : secret "maas-api-e2e-shared-b-w3-117711-serving-cert" not found Sep 11 11:53:37.498892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.498875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c8284-f727-43ae-854e-c60c03b0438b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.500699 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.500679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:37.512921 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:37.512899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5dmg\" (UniqueName: \"kubernetes.io/projected/2a8c8284-f727-43ae-854e-c60c03b0438b-kube-api-access-v5dmg\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:38.003367 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:38.003329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:38.005945 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:38.005921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:38.228180 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:38.228141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:38.567893 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:38.567870 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6"] Sep 11 11:53:38.569734 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:53:38.569709 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8c8284_f727_43ae_854e_c60c03b0438b.slice/crio-5bbc843bb83ed79177e7477e80207d315270f8a022b0ee75b92e994ef74b3bb3 WatchSource:0}: Error finding container 5bbc843bb83ed79177e7477e80207d315270f8a022b0ee75b92e994ef74b3bb3: Status 404 returned error can't find the container with id 5bbc843bb83ed79177e7477e80207d315270f8a022b0ee75b92e994ef74b3bb3 Sep 11 11:53:38.571126 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:38.571103 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 11:53:39.473993 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.473921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" event={"ID":"2a8c8284-f727-43ae-854e-c60c03b0438b","Type":"ContainerStarted","Data":"5bbc843bb83ed79177e7477e80207d315270f8a022b0ee75b92e994ef74b3bb3"} Sep 11 11:53:39.802393 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.802320 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk"] Sep 11 11:53:39.806298 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.806277 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.809013 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.808831 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6daa53-openshift-default-dockercfg-lcmff\"" Sep 11 11:53:39.824437 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.824413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk"] Sep 11 11:53:39.924543 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924491 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-certs\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.924723 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-envoy\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.924723 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79ab351-95a0-43b8-9110-623194a4feaa-istiod-ca-cert\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.924723 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-socket\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.924895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vgp\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-kube-api-access-m7vgp\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.924895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-data\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.924895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924796 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-istio-token\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.924895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79ab351-95a0-43b8-9110-623194a4feaa-istio-podinfo\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:39.925095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:39.924941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-credential-socket\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.025709 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.025662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-socket\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.025709 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.025697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vgp\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-kube-api-access-m7vgp\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.025937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.025833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-data\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.025937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.025877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-istio-token\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.025937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.025895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79ab351-95a0-43b8-9110-623194a4feaa-istio-podinfo\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026142 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.025998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-credential-socket\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026142 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026083 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-socket\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026142 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-certs\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-envoy\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-data\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79ab351-95a0-43b8-9110-623194a4feaa-istiod-ca-cert\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026457 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-credential-socket\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026457 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-certs\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.026994 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.026834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79ab351-95a0-43b8-9110-623194a4feaa-istiod-ca-cert\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.028667 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.028644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-envoy\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.028885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.028862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79ab351-95a0-43b8-9110-623194a4feaa-istio-podinfo\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.036629 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.036609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-istio-token\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.036960 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.036933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vgp\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-kube-api-access-m7vgp\") pod \"e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:40.120494 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:40.120413 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:41.185296 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.185263 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk"] Sep 11 11:53:41.186465 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:53:41.186438 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79ab351_95a0_43b8_9110_623194a4feaa.slice/crio-b3eb0d4a2641d711425c0669f31bf5f143b1208fc91fa8b6cd65fe0b6f6e388c WatchSource:0}: Error finding container b3eb0d4a2641d711425c0669f31bf5f143b1208fc91fa8b6cd65fe0b6f6e388c: Status 404 returned error can't find the container with id b3eb0d4a2641d711425c0669f31bf5f143b1208fc91fa8b6cd65fe0b6f6e388c Sep 11 11:53:41.189327 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.189291 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:41.189439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.189354 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:41.189439 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.189384 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:41.488605 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.488494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" event={"ID":"2a8c8284-f727-43ae-854e-c60c03b0438b","Type":"ContainerStarted","Data":"f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af"} Sep 11 11:53:41.488754 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.488671 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:41.490432 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.490408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" event={"ID":"d79ab351-95a0-43b8-9110-623194a4feaa","Type":"ContainerStarted","Data":"98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743"} Sep 11 11:53:41.490575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.490452 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" event={"ID":"d79ab351-95a0-43b8-9110-623194a4feaa","Type":"ContainerStarted","Data":"b3eb0d4a2641d711425c0669f31bf5f143b1208fc91fa8b6cd65fe0b6f6e388c"} Sep 11 11:53:41.542372 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.542329 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" podStartSLOduration=2.149269591 podStartE2EDuration="4.542314673s" totalImagesPullingTime="2.393045082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:53:41.515427114 +0000 UTC m=+1183.150020247" watchObservedRunningTime="2026-09-11 11:53:41.542314673 +0000 UTC m=+1183.176907806" Sep 11 11:53:41.542653 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:41.542621 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podStartSLOduration=2.542615803 podStartE2EDuration="2.542615803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:53:41.541729887 +0000 UTC m=+1183.176323032" watchObservedRunningTime="2026-09-11 11:53:41.542615803 +0000 UTC m=+1183.177208936" Sep 11 11:53:42.120712 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:42.120677 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:42.126109 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:42.126083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:42.496937 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:42.496894 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:42.497829 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:42.497810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:53:43.212299 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:43.212266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:53:47.502061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:47.502000 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 11:53:51.487264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:51.487219 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 11:53:51.487774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:51.487288 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 11:53:54.597186 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:54.597150 2580 generic.go:356] "Generic (PLEG): container finished" podID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerID="1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748" exitCode=0 Sep 11 11:53:54.597643 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:54.597222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" event={"ID":"e3f1a741-2564-486d-910c-e2c33bf91e94","Type":"ContainerDied","Data":"1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748"} Sep 11 11:53:54.597643 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:54.597615 2580 scope.go:117] "RemoveContainer" containerID="1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748" Sep 11 11:53:54.598682 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:54.598651 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:54.598772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:54.598717 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:54.598772 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:54.598748 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:53:55.098208 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:55.098164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:53:55.609941 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:55.609906 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" event={"ID":"e3f1a741-2564-486d-910c-e2c33bf91e94","Type":"ContainerStarted","Data":"dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5"} Sep 11 11:53:55.904670 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:55.904579 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:53:55.904670 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:55.904635 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 11:53:55.904951 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:55.904694 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 11:53:56.904388 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:56.904349 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 11:53:56.904890 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:56.904403 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 11:53:57.905103 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:57.905065 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 11:53:57.905560 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:57.905119 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 11:53:58.538178 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:58.538140 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 11:53:58.538343 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:58.538188 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 11:53:58.904231 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:58.904196 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 11:53:58.904516 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:58.904480 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 11:53:58.925924 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:58.925899 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:53:58.929296 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:58.929274 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:53:58.986911 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:58.986877 2580 scope.go:117] "RemoveContainer" containerID="00e75e8467b2307ec0144da0bf1acb24d6e0b2b5f15cd2da443996363b1e59aa" Sep 11 11:53:59.904275 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:59.904239 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 11:53:59.904447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:53:59.904296 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 11:54:00.640991 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.640913 2580 generic.go:356] "Generic (PLEG): container finished" podID="d79ab351-95a0-43b8-9110-623194a4feaa" containerID="98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743" exitCode=0 Sep 11 11:54:00.641388 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.640995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" event={"ID":"d79ab351-95a0-43b8-9110-623194a4feaa","Type":"ContainerDied","Data":"98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743"} Sep 11 11:54:00.641430 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.641406 2580 scope.go:117] "RemoveContainer" containerID="98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743" Sep 11 11:54:00.642556 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.642528 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:54:00.642609 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.642597 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:54:00.642648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.642625 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:54:00.904474 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.904439 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 11:54:00.904675 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:00.904495 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 11:54:01.649418 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:01.649384 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" event={"ID":"d79ab351-95a0-43b8-9110-623194a4feaa","Type":"ContainerStarted","Data":"f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5"} Sep 11 11:54:01.909062 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:01.908971 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:54:01.909292 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:01.909275 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:54:01.909936 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:01.909918 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 11:54:02.120839 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:02.120800 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:54:02.121047 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:02.121009 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 11:54:02.121117 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:02.121093 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 11:54:03.121765 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:03.121729 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 11:54:03.122194 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:03.121821 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 11:54:04.121558 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:04.121519 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 11:54:04.121737 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:04.121587 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 11:54:05.121326 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:05.121291 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 11:54:05.121785 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:05.121360 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 11:54:06.120895 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:06.120859 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 11:54:06.121094 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:06.120913 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 11:54:07.121446 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:07.121407 2580 patch_prober.go:28] interesting pod/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 11:54:07.121844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:07.121472 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 11:54:08.189984 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:08.189949 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:54:08.190407 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:08.190364 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:54:08.191153 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:08.191133 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 11:54:14.162796 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.162762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8011-1d6abe"] Sep 11 11:54:14.168762 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.168743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:14.171909 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.171885 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5xz2c\"" Sep 11 11:54:14.177509 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.177481 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-1d6abe"] Sep 11 11:54:14.320922 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.320874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd22d\" (UniqueName: \"kubernetes.io/projected/49b89a1a-6404-4fd6-9d49-cc9f34059da1-kube-api-access-qd22d\") pod \"test-curl-8011-1d6abe\" (UID: \"49b89a1a-6404-4fd6-9d49-cc9f34059da1\") " pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:14.421460 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.421381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd22d\" (UniqueName: \"kubernetes.io/projected/49b89a1a-6404-4fd6-9d49-cc9f34059da1-kube-api-access-qd22d\") pod \"test-curl-8011-1d6abe\" (UID: \"49b89a1a-6404-4fd6-9d49-cc9f34059da1\") " pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:14.430447 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.430421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd22d\" (UniqueName: \"kubernetes.io/projected/49b89a1a-6404-4fd6-9d49-cc9f34059da1-kube-api-access-qd22d\") pod \"test-curl-8011-1d6abe\" (UID: \"49b89a1a-6404-4fd6-9d49-cc9f34059da1\") " pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:14.479708 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.479684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:14.749923 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:14.749886 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-1d6abe"] Sep 11 11:54:14.751415 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:54:14.751387 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49b89a1a_6404_4fd6_9d49_cc9f34059da1.slice/crio-03e34fb54409fc89bc8dcea4f6a8a624790295224220ea71a0e1b83fa078f8bf WatchSource:0}: Error finding container 03e34fb54409fc89bc8dcea4f6a8a624790295224220ea71a0e1b83fa078f8bf: Status 404 returned error can't find the container with id 03e34fb54409fc89bc8dcea4f6a8a624790295224220ea71a0e1b83fa078f8bf Sep 11 11:54:15.741892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:15.741858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-1d6abe" event={"ID":"49b89a1a-6404-4fd6-9d49-cc9f34059da1","Type":"ContainerStarted","Data":"2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35"} Sep 11 11:54:15.742335 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:15.741901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-1d6abe" event={"ID":"49b89a1a-6404-4fd6-9d49-cc9f34059da1","Type":"ContainerStarted","Data":"03e34fb54409fc89bc8dcea4f6a8a624790295224220ea71a0e1b83fa078f8bf"} Sep 11 11:54:15.758867 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:15.758819 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8011-1d6abe" podStartSLOduration=1.139341452 podStartE2EDuration="1.758803219s" totalImagesPullingTime="619.461767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:15.757130574 +0000 UTC m=+1217.391723707" watchObservedRunningTime="2026-09-11 11:54:15.758803219 +0000 UTC m=+1217.393396406" Sep 11 11:54:16.334477 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.334431 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8011-1d6abe"] Sep 11 11:54:16.338248 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.338217 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8011-1d6abe"] Sep 11 11:54:16.445576 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.445547 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8011-f414f1"] Sep 11 11:54:16.445946 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.445934 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b89a1a-6404-4fd6-9d49-cc9f34059da1" containerName="test-curl-8011-1d6abe" Sep 11 11:54:16.445993 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.445947 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b89a1a-6404-4fd6-9d49-cc9f34059da1" containerName="test-curl-8011-1d6abe" Sep 11 11:54:16.446028 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.446019 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b89a1a-6404-4fd6-9d49-cc9f34059da1" containerName="test-curl-8011-1d6abe" Sep 11 11:54:16.449183 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.449159 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:16.454741 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.454718 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-f414f1"] Sep 11 11:54:16.541794 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.541762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snh2k\" (UniqueName: \"kubernetes.io/projected/ad24ea57-70d5-42de-8a4b-f81751147843-kube-api-access-snh2k\") pod \"test-curl-8011-f414f1\" (UID: \"ad24ea57-70d5-42de-8a4b-f81751147843\") " pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:16.642774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.642745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snh2k\" (UniqueName: \"kubernetes.io/projected/ad24ea57-70d5-42de-8a4b-f81751147843-kube-api-access-snh2k\") pod \"test-curl-8011-f414f1\" (UID: \"ad24ea57-70d5-42de-8a4b-f81751147843\") " pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:16.650615 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.650585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snh2k\" (UniqueName: \"kubernetes.io/projected/ad24ea57-70d5-42de-8a4b-f81751147843-kube-api-access-snh2k\") pod \"test-curl-8011-f414f1\" (UID: \"ad24ea57-70d5-42de-8a4b-f81751147843\") " pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:16.747860 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:16.747778 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8011-1d6abe" container="test-curl-8011-1d6abe" Sep 11 11:54:16.760695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:16.760672 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:17.033647 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:17.033624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-f414f1"] Sep 11 11:54:17.035677 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:54:17.035649 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad24ea57_70d5_42de_8a4b_f81751147843.slice/crio-fae33cb244c92927c3d3d2ab89e853fc846fef3668a462387201b7568de19412 WatchSource:0}: Error finding container fae33cb244c92927c3d3d2ab89e853fc846fef3668a462387201b7568de19412: Status 404 returned error can't find the container with id fae33cb244c92927c3d3d2ab89e853fc846fef3668a462387201b7568de19412 Sep 11 11:54:17.754548 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:17.754506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-f414f1" event={"ID":"ad24ea57-70d5-42de-8a4b-f81751147843","Type":"ContainerStarted","Data":"c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb"} Sep 11 11:54:17.755007 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:17.754556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-f414f1" event={"ID":"ad24ea57-70d5-42de-8a4b-f81751147843","Type":"ContainerStarted","Data":"fae33cb244c92927c3d3d2ab89e853fc846fef3668a462387201b7568de19412"} Sep 11 11:54:17.755007 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:17.754580 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8011-1d6abe" podUID="49b89a1a-6404-4fd6-9d49-cc9f34059da1" containerName="test-curl-8011-1d6abe" containerID="cri-o://2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35" gracePeriod=2 Sep 11 11:54:17.771211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:17.771166 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8011-f414f1" podStartSLOduration=1.771153066 podStartE2EDuration="1.771153066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:17.769019782 +0000 UTC m=+1219.403612914" watchObservedRunningTime="2026-09-11 11:54:17.771153066 +0000 UTC m=+1219.405746198" Sep 11 11:54:18.329264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.329231 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8011-f414f1"] Sep 11 11:54:18.333474 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.333443 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8011-f414f1"] Sep 11 11:54:18.436857 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.436830 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8011-3653ff"] Sep 11 11:54:18.437242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.437228 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad24ea57-70d5-42de-8a4b-f81751147843" containerName="test-curl-8011-f414f1" Sep 11 11:54:18.437292 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.437244 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad24ea57-70d5-42de-8a4b-f81751147843" containerName="test-curl-8011-f414f1" Sep 11 11:54:18.437328 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.437314 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad24ea57-70d5-42de-8a4b-f81751147843" containerName="test-curl-8011-f414f1" Sep 11 11:54:18.440414 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.440401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:18.448543 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.447953 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-3653ff"] Sep 11 11:54:18.561556 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.561521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbrls\" (UniqueName: \"kubernetes.io/projected/3f7f9ff2-7150-482c-acf2-031fd2b69f79-kube-api-access-rbrls\") pod \"test-curl-8011-3653ff\" (UID: \"3f7f9ff2-7150-482c-acf2-031fd2b69f79\") " pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:18.662599 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.662564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbrls\" (UniqueName: \"kubernetes.io/projected/3f7f9ff2-7150-482c-acf2-031fd2b69f79-kube-api-access-rbrls\") pod \"test-curl-8011-3653ff\" (UID: \"3f7f9ff2-7150-482c-acf2-031fd2b69f79\") " pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:18.671679 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.671651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbrls\" (UniqueName: \"kubernetes.io/projected/3f7f9ff2-7150-482c-acf2-031fd2b69f79-kube-api-access-rbrls\") pod \"test-curl-8011-3653ff\" (UID: \"3f7f9ff2-7150-482c-acf2-031fd2b69f79\") " pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:18.750662 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:18.750597 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:18.761406 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:18.761383 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8011-f414f1" container="test-curl-8011-f414f1" Sep 11 11:54:19.029166 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:19.029141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-3653ff"] Sep 11 11:54:19.030629 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:54:19.030603 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f7f9ff2_7150_482c_acf2_031fd2b69f79.slice/crio-0d5d6c8a0ccb2870f95ef35349cca91fa27e076d53c71867ac8b40f4d7313df3 WatchSource:0}: Error finding container 0d5d6c8a0ccb2870f95ef35349cca91fa27e076d53c71867ac8b40f4d7313df3: Status 404 returned error can't find the container with id 0d5d6c8a0ccb2870f95ef35349cca91fa27e076d53c71867ac8b40f4d7313df3 Sep 11 11:54:19.768665 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:19.768635 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-3653ff" event={"ID":"3f7f9ff2-7150-482c-acf2-031fd2b69f79","Type":"ContainerStarted","Data":"94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2"} Sep 11 11:54:19.769073 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:19.768677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-3653ff" event={"ID":"3f7f9ff2-7150-482c-acf2-031fd2b69f79","Type":"ContainerStarted","Data":"0d5d6c8a0ccb2870f95ef35349cca91fa27e076d53c71867ac8b40f4d7313df3"} Sep 11 11:54:19.769073 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:19.768903 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8011-f414f1" podUID="ad24ea57-70d5-42de-8a4b-f81751147843" containerName="test-curl-8011-f414f1" containerID="cri-o://c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb" gracePeriod=2 Sep 11 11:54:19.785182 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:19.785143 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8011-3653ff" podStartSLOduration=1.785129105 podStartE2EDuration="1.785129105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:19.782516605 +0000 UTC m=+1221.417109740" watchObservedRunningTime="2026-09-11 11:54:19.785129105 +0000 UTC m=+1221.419722239" Sep 11 11:54:19.943633 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:19.943601 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:20.077523 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.077432 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd22d\" (UniqueName: \"kubernetes.io/projected/49b89a1a-6404-4fd6-9d49-cc9f34059da1-kube-api-access-qd22d\") pod \"49b89a1a-6404-4fd6-9d49-cc9f34059da1\" (UID: \"49b89a1a-6404-4fd6-9d49-cc9f34059da1\") " Sep 11 11:54:20.079719 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.079689 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b89a1a-6404-4fd6-9d49-cc9f34059da1-kube-api-access-qd22d" (OuterVolumeSpecName: "kube-api-access-qd22d") pod "49b89a1a-6404-4fd6-9d49-cc9f34059da1" (UID: "49b89a1a-6404-4fd6-9d49-cc9f34059da1"). InnerVolumeSpecName "kube-api-access-qd22d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:54:20.178277 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.178252 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd22d\" (UniqueName: \"kubernetes.io/projected/49b89a1a-6404-4fd6-9d49-cc9f34059da1-kube-api-access-qd22d\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:54:20.338702 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.338615 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8011-3653ff"] Sep 11 11:54:20.345073 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.345022 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8011-3653ff"] Sep 11 11:54:20.451066 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.451022 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8011-e21651"] Sep 11 11:54:20.451452 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.451439 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7f9ff2-7150-482c-acf2-031fd2b69f79" containerName="test-curl-8011-3653ff" Sep 11 11:54:20.451499 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.451454 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7f9ff2-7150-482c-acf2-031fd2b69f79" containerName="test-curl-8011-3653ff" Sep 11 11:54:20.451535 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.451520 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7f9ff2-7150-482c-acf2-031fd2b69f79" containerName="test-curl-8011-3653ff" Sep 11 11:54:20.454577 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.454562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:20.462342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.461982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-e21651"] Sep 11 11:54:20.582958 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.582923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcpq\" (UniqueName: \"kubernetes.io/projected/8435913c-05a8-4391-a557-be786f2ee98f-kube-api-access-tqcpq\") pod \"test-curl-8011-e21651\" (UID: \"8435913c-05a8-4391-a557-be786f2ee98f\") " pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:20.683519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.683482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcpq\" (UniqueName: \"kubernetes.io/projected/8435913c-05a8-4391-a557-be786f2ee98f-kube-api-access-tqcpq\") pod \"test-curl-8011-e21651\" (UID: \"8435913c-05a8-4391-a557-be786f2ee98f\") " pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:20.692094 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.692068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcpq\" (UniqueName: \"kubernetes.io/projected/8435913c-05a8-4391-a557-be786f2ee98f-kube-api-access-tqcpq\") pod \"test-curl-8011-e21651\" (UID: \"8435913c-05a8-4391-a557-be786f2ee98f\") " pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:20.765687 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.765652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:20.776017 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.775992 2580 generic.go:356] "Generic (PLEG): container finished" podID="49b89a1a-6404-4fd6-9d49-cc9f34059da1" containerID="2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35" exitCode=137 Sep 11 11:54:20.776361 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.776084 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:20.776361 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.776082 2580 scope.go:117] "RemoveContainer" containerID="2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35" Sep 11 11:54:20.776483 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:20.776457 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8011-3653ff" container="test-curl-8011-3653ff" Sep 11 11:54:20.797518 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:20.797485 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8011-1d6abe\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="49b89a1a-6404-4fd6-9d49-cc9f34059da1" pod="opendatahub/test-curl-8011-1d6abe" Sep 11 11:54:20.797946 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.797923 2580 scope.go:117] "RemoveContainer" containerID="2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35" Sep 11 11:54:20.798329 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:20.798305 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35\": container with ID starting with 2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35 not found: ID does not exist" containerID="2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35" Sep 11 11:54:20.798412 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.798343 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35"} err="failed to get container status \"2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35\": rpc error: code = NotFound desc = could not find container \"2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35\": container with ID starting with 2a870ba55c07a942c7673da46e278e8458ba3f732d1d05c25ba94fcf00d26e35 not found: ID does not exist" Sep 11 11:54:20.873567 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:20.873527 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b89a1a-6404-4fd6-9d49-cc9f34059da1" path="/var/lib/kubelet/pods/49b89a1a-6404-4fd6-9d49-cc9f34059da1/volumes" Sep 11 11:54:21.040056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:21.040017 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8011-e21651"] Sep 11 11:54:21.041796 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:54:21.041771 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8435913c_05a8_4391_a557_be786f2ee98f.slice/crio-b6938138fd55f4d35b14f144db145a3b785c035e540db4302f09b5fe36457650 WatchSource:0}: Error finding container b6938138fd55f4d35b14f144db145a3b785c035e540db4302f09b5fe36457650: Status 404 returned error can't find the container with id b6938138fd55f4d35b14f144db145a3b785c035e540db4302f09b5fe36457650 Sep 11 11:54:21.785492 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:21.785465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-e21651" event={"ID":"8435913c-05a8-4391-a557-be786f2ee98f","Type":"ContainerStarted","Data":"1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95"} Sep 11 11:54:21.785836 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:21.785505 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8011-e21651" event={"ID":"8435913c-05a8-4391-a557-be786f2ee98f","Type":"ContainerStarted","Data":"b6938138fd55f4d35b14f144db145a3b785c035e540db4302f09b5fe36457650"} Sep 11 11:54:21.786480 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:21.786457 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8011-3653ff" podUID="3f7f9ff2-7150-482c-acf2-031fd2b69f79" containerName="test-curl-8011-3653ff" containerID="cri-o://94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2" gracePeriod=2 Sep 11 11:54:21.802696 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:21.802655 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8011-e21651" podStartSLOduration=1.8026410579999999 podStartE2EDuration="1.802641058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:21.801678509 +0000 UTC m=+1223.436271645" watchObservedRunningTime="2026-09-11 11:54:21.802641058 +0000 UTC m=+1223.437234236" Sep 11 11:54:21.938530 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:21.938501 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:22.098347 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.098263 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snh2k\" (UniqueName: \"kubernetes.io/projected/ad24ea57-70d5-42de-8a4b-f81751147843-kube-api-access-snh2k\") pod \"ad24ea57-70d5-42de-8a4b-f81751147843\" (UID: \"ad24ea57-70d5-42de-8a4b-f81751147843\") " Sep 11 11:54:22.100424 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.100389 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad24ea57-70d5-42de-8a4b-f81751147843-kube-api-access-snh2k" (OuterVolumeSpecName: "kube-api-access-snh2k") pod "ad24ea57-70d5-42de-8a4b-f81751147843" (UID: "ad24ea57-70d5-42de-8a4b-f81751147843"). InnerVolumeSpecName "kube-api-access-snh2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:54:22.198906 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.198878 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snh2k\" (UniqueName: \"kubernetes.io/projected/ad24ea57-70d5-42de-8a4b-f81751147843-kube-api-access-snh2k\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:54:22.362282 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.362198 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8011-e21651"] Sep 11 11:54:22.371452 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.371422 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8011-e21651"] Sep 11 11:54:22.794392 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.794357 2580 generic.go:356] "Generic (PLEG): container finished" podID="ad24ea57-70d5-42de-8a4b-f81751147843" containerID="c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb" exitCode=137 Sep 11 11:54:22.794392 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.794395 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:22.794861 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.794462 2580 scope.go:117] "RemoveContainer" containerID="c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb" Sep 11 11:54:22.794861 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:22.794667 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8011-e21651" container="test-curl-8011-e21651" Sep 11 11:54:22.803965 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.803949 2580 scope.go:117] "RemoveContainer" containerID="c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb" Sep 11 11:54:22.804235 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:22.804215 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb\": container with ID starting with c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb not found: ID does not exist" containerID="c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb" Sep 11 11:54:22.804322 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.804246 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb"} err="failed to get container status \"c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb\": rpc error: code = NotFound desc = could not find container \"c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb\": container with ID starting with c17f9311c4bace71480adb2c069d62c65bb10e100d5d861bf9ab251a6cb1e0eb not found: ID does not exist" Sep 11 11:54:22.804394 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:22.804375 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8011-f414f1\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="ad24ea57-70d5-42de-8a4b-f81751147843" pod="opendatahub/test-curl-8011-f414f1" Sep 11 11:54:22.873593 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:22.873568 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad24ea57-70d5-42de-8a4b-f81751147843" path="/var/lib/kubelet/pods/ad24ea57-70d5-42de-8a4b-f81751147843/volumes" Sep 11 11:54:23.216389 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.216352 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf"] Sep 11 11:54:23.216779 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.216766 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8435913c-05a8-4391-a557-be786f2ee98f" containerName="test-curl-8011-e21651" Sep 11 11:54:23.216819 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.216781 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8435913c-05a8-4391-a557-be786f2ee98f" containerName="test-curl-8011-e21651" Sep 11 11:54:23.216870 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.216843 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8435913c-05a8-4391-a557-be786f2ee98f" containerName="test-curl-8011-e21651" Sep 11 11:54:23.221215 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.221200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.225501 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.225479 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-307b7f39-openshift-default-dockercfg-s9ts9\"" Sep 11 11:54:23.242333 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.242307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf"] Sep 11 11:54:23.309704 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-envoy\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.309886 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-podinfo\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.309886 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ml2t\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-kube-api-access-8ml2t\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.309886 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-credential-socket\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.309886 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-socket\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.310161 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-certs\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.310161 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istiod-ca-cert\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.310161 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.309984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-token\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.310161 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.310010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-data\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.410762 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.410728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-envoy\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.410762 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.410765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-podinfo\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.410981 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.410801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ml2t\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-kube-api-access-8ml2t\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.410981 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.410826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-credential-socket\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.410981 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.410869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-socket\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.410981 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.410898 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-certs\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.411700 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.411524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istiod-ca-cert\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.411700 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.411620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-token\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.411700 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.411619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-credential-socket\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.411979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.411721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-certs\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.411979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.411750 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-socket\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.411979 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.411865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-data\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.412241 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.412222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-data\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.412701 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.412674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istiod-ca-cert\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.414377 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.414351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-podinfo\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.419539 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.414788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-envoy\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.419650 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.419621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ml2t\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-kube-api-access-8ml2t\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.419975 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.419955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-token\") pod \"e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.531730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.531639 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:23.802408 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.802379 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8011-e21651" podUID="8435913c-05a8-4391-a557-be786f2ee98f" containerName="test-curl-8011-e21651" containerID="cri-o://1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95" gracePeriod=2 Sep 11 11:54:23.817823 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.817796 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf"] Sep 11 11:54:23.829919 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:54:23.829895 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20e2c8f_0764_480c_8842_2e3ed53a53fc.slice/crio-c6b90e684153794d72d0f19eff07f38412f975acec38e536fe5fbcfc029b60ee WatchSource:0}: Error finding container c6b90e684153794d72d0f19eff07f38412f975acec38e536fe5fbcfc029b60ee: Status 404 returned error can't find the container with id c6b90e684153794d72d0f19eff07f38412f975acec38e536fe5fbcfc029b60ee Sep 11 11:54:23.832131 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.832099 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:54:23.832207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.832175 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:54:23.832207 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.832203 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:54:23.956882 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:23.956859 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:24.117379 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.117275 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbrls\" (UniqueName: \"kubernetes.io/projected/3f7f9ff2-7150-482c-acf2-031fd2b69f79-kube-api-access-rbrls\") pod \"3f7f9ff2-7150-482c-acf2-031fd2b69f79\" (UID: \"3f7f9ff2-7150-482c-acf2-031fd2b69f79\") " Sep 11 11:54:24.119522 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.119490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7f9ff2-7150-482c-acf2-031fd2b69f79-kube-api-access-rbrls" (OuterVolumeSpecName: "kube-api-access-rbrls") pod "3f7f9ff2-7150-482c-acf2-031fd2b69f79" (UID: "3f7f9ff2-7150-482c-acf2-031fd2b69f79"). InnerVolumeSpecName "kube-api-access-rbrls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:54:24.218435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.218408 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbrls\" (UniqueName: \"kubernetes.io/projected/3f7f9ff2-7150-482c-acf2-031fd2b69f79-kube-api-access-rbrls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:54:24.809774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.809686 2580 generic.go:356] "Generic (PLEG): container finished" podID="3f7f9ff2-7150-482c-acf2-031fd2b69f79" containerID="94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2" exitCode=137 Sep 11 11:54:24.809774 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.809730 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:24.810264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.809794 2580 scope.go:117] "RemoveContainer" containerID="94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2" Sep 11 11:54:24.811866 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.811843 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" event={"ID":"e20e2c8f-0764-480c-8842-2e3ed53a53fc","Type":"ContainerStarted","Data":"26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384"} Sep 11 11:54:24.812001 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.811897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" event={"ID":"e20e2c8f-0764-480c-8842-2e3ed53a53fc","Type":"ContainerStarted","Data":"c6b90e684153794d72d0f19eff07f38412f975acec38e536fe5fbcfc029b60ee"} Sep 11 11:54:24.819366 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.819343 2580 scope.go:117] "RemoveContainer" containerID="94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2" Sep 11 11:54:24.819626 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:24.819611 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2\": container with ID starting with 94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2 not found: ID does not exist" containerID="94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2" Sep 11 11:54:24.819672 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.819634 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2"} err="failed to get container status \"94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2\": rpc error: code = NotFound desc = could not find container \"94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2\": container with ID starting with 94c491e6be8c17e06de594ddcfc64f170c41b24730332ea6e71e2cd60e672bc2 not found: ID does not exist" Sep 11 11:54:24.849245 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:24.849213 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8011-3653ff\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="3f7f9ff2-7150-482c-acf2-031fd2b69f79" pod="opendatahub/test-curl-8011-3653ff" Sep 11 11:54:24.849640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.849602 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" podStartSLOduration=1.849590295 podStartE2EDuration="1.849590295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:24.847271699 +0000 UTC m=+1226.481864843" watchObservedRunningTime="2026-09-11 11:54:24.849590295 +0000 UTC m=+1226.484183428" Sep 11 11:54:24.872630 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:24.872607 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f7f9ff2-7150-482c-acf2-031fd2b69f79" path="/var/lib/kubelet/pods/3f7f9ff2-7150-482c-acf2-031fd2b69f79/volumes" Sep 11 11:54:25.532814 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:25.532782 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:25.537825 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:25.537802 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:25.819166 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:25.819144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:25.820216 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:25.820193 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:54:25.995091 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:25.995068 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:26.138361 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.138281 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqcpq\" (UniqueName: \"kubernetes.io/projected/8435913c-05a8-4391-a557-be786f2ee98f-kube-api-access-tqcpq\") pod \"8435913c-05a8-4391-a557-be786f2ee98f\" (UID: \"8435913c-05a8-4391-a557-be786f2ee98f\") " Sep 11 11:54:26.140481 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.140452 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8435913c-05a8-4391-a557-be786f2ee98f-kube-api-access-tqcpq" (OuterVolumeSpecName: "kube-api-access-tqcpq") pod "8435913c-05a8-4391-a557-be786f2ee98f" (UID: "8435913c-05a8-4391-a557-be786f2ee98f"). InnerVolumeSpecName "kube-api-access-tqcpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:54:26.239364 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.239334 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqcpq\" (UniqueName: \"kubernetes.io/projected/8435913c-05a8-4391-a557-be786f2ee98f-kube-api-access-tqcpq\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:54:26.826262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.826224 2580 generic.go:356] "Generic (PLEG): container finished" podID="8435913c-05a8-4391-a557-be786f2ee98f" containerID="1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95" exitCode=137 Sep 11 11:54:26.826262 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.826265 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:26.826754 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.826325 2580 scope.go:117] "RemoveContainer" containerID="1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95" Sep 11 11:54:26.838794 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.838774 2580 scope.go:117] "RemoveContainer" containerID="1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95" Sep 11 11:54:26.839130 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:26.839109 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95\": container with ID starting with 1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95 not found: ID does not exist" containerID="1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95" Sep 11 11:54:26.839229 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.839137 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95"} err="failed to get container status \"1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95\": rpc error: code = NotFound desc = could not find container \"1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95\": container with ID starting with 1406f754cdfdd4dfab8b18e7cc42b5fcbcccbba80d82c4f6d01856e3f6ee4e95 not found: ID does not exist" Sep 11 11:54:26.839803 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:54:26.839771 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8011-e21651\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="8435913c-05a8-4391-a557-be786f2ee98f" pod="opendatahub/test-curl-8011-e21651" Sep 11 11:54:26.873299 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:26.873275 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8435913c-05a8-4391-a557-be786f2ee98f" path="/var/lib/kubelet/pods/8435913c-05a8-4391-a557-be786f2ee98f/volumes" Sep 11 11:54:27.917096 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:27.917064 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh"] Sep 11 11:54:27.922168 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:27.922145 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:27.924466 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:27.924443 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-307b7f39-serving-cert\"" Sep 11 11:54:27.934855 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:27.934832 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh"] Sep 11 11:54:28.057248 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.057215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-maas-api-tls\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.057248 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.057247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-metrics-certs\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.057512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.057420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6439689a-bc56-45f3-acf4-5d32da43510d-openshift-service-ca\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.057512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.057458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtv5q\" (UniqueName: \"kubernetes.io/projected/6439689a-bc56-45f3-acf4-5d32da43510d-kube-api-access-xtv5q\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.158099 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.158064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6439689a-bc56-45f3-acf4-5d32da43510d-openshift-service-ca\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.158242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.158107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtv5q\" (UniqueName: \"kubernetes.io/projected/6439689a-bc56-45f3-acf4-5d32da43510d-kube-api-access-xtv5q\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.158242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.158206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-maas-api-tls\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.158242 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.158233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-metrics-certs\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.158658 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.158638 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6439689a-bc56-45f3-acf4-5d32da43510d-openshift-service-ca\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.160802 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.160781 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-maas-api-tls\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.161085 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.161060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-metrics-certs\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.166572 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.166545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtv5q\" (UniqueName: \"kubernetes.io/projected/6439689a-bc56-45f3-acf4-5d32da43510d-kube-api-access-xtv5q\") pod \"maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.233771 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.233694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:28.509175 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.509148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh"] Sep 11 11:54:28.510921 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:54:28.510889 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6439689a_bc56_45f3_acf4_5d32da43510d.slice/crio-7c69b82049b1521b74a3f3ddf72afa3bce75ec0ede9a9b206e55529441f4708d WatchSource:0}: Error finding container 7c69b82049b1521b74a3f3ddf72afa3bce75ec0ede9a9b206e55529441f4708d: Status 404 returned error can't find the container with id 7c69b82049b1521b74a3f3ddf72afa3bce75ec0ede9a9b206e55529441f4708d Sep 11 11:54:28.840206 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:28.840125 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" event={"ID":"6439689a-bc56-45f3-acf4-5d32da43510d","Type":"ContainerStarted","Data":"7c69b82049b1521b74a3f3ddf72afa3bce75ec0ede9a9b206e55529441f4708d"} Sep 11 11:54:29.847429 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:29.847392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" event={"ID":"6439689a-bc56-45f3-acf4-5d32da43510d","Type":"ContainerStarted","Data":"5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab"} Sep 11 11:54:29.847817 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:29.847523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:29.865601 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:29.865553 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" podStartSLOduration=2.49214856 podStartE2EDuration="2.865537905s" totalImagesPullingTime="373.389345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:29.864719627 +0000 UTC m=+1231.499312764" watchObservedRunningTime="2026-09-11 11:54:29.865537905 +0000 UTC m=+1231.500131039" Sep 11 11:54:35.294129 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:35.294086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:54:35.861696 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:35.861664 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:54:39.916632 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.916602 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2"] Sep 11 11:54:39.920410 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.920394 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:39.923373 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.923348 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-272c5e\"/\"kube-root-ca.crt\"" Sep 11 11:54:39.923518 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.923348 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-272c5e\"/\"openshift-service-ca.crt\"" Sep 11 11:54:39.923518 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.923383 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-272c5e\"/\"default-dockercfg-x55lf\"" Sep 11 11:54:39.923518 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.923384 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-272c5e\"/\"auth-test-model-272c5e-kserve-self-signed-certs\"" Sep 11 11:54:39.929858 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.929840 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2"] Sep 11 11:54:39.962416 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.962383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-dshm\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:39.962553 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.962424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df385b57-ee3a-49f9-a3ed-b201b1847f11-tls-certs\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:39.962553 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.962451 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-tmp-dir\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:39.962648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.962567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-model-cache\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:39.962648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.962604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xs7k\" (UniqueName: \"kubernetes.io/projected/df385b57-ee3a-49f9-a3ed-b201b1847f11-kube-api-access-4xs7k\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:39.962648 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:39.962637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-home\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.063634 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.063594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-model-cache\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.063787 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.063646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xs7k\" (UniqueName: \"kubernetes.io/projected/df385b57-ee3a-49f9-a3ed-b201b1847f11-kube-api-access-4xs7k\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.063787 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.063684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-home\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.063891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.063796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-dshm\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.063891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.063818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df385b57-ee3a-49f9-a3ed-b201b1847f11-tls-certs\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.063891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.063841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-tmp-dir\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.064080 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.064057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-model-cache\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.064152 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.064129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-home\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.064247 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.064227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-tmp-dir\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.066264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.066234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-dshm\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.066427 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.066412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df385b57-ee3a-49f9-a3ed-b201b1847f11-tls-certs\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.071996 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.071971 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xs7k\" (UniqueName: \"kubernetes.io/projected/df385b57-ee3a-49f9-a3ed-b201b1847f11-kube-api-access-4xs7k\") pod \"auth-test-model-272c5e-kserve-6577f549f5-wxss2\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.232322 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.232231 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:40.509095 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.509069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2"] Sep 11 11:54:40.510691 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:54:40.510645 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf385b57_ee3a_49f9_a3ed_b201b1847f11.slice/crio-312f2ab05d19adcbac6d5fa6fad870c9ffba1fd6024021f9864cb6ebbe245fd2 WatchSource:0}: Error finding container 312f2ab05d19adcbac6d5fa6fad870c9ffba1fd6024021f9864cb6ebbe245fd2: Status 404 returned error can't find the container with id 312f2ab05d19adcbac6d5fa6fad870c9ffba1fd6024021f9864cb6ebbe245fd2 Sep 11 11:54:40.921417 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.921387 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" event={"ID":"df385b57-ee3a-49f9-a3ed-b201b1847f11","Type":"ContainerStarted","Data":"6f47d00b11b0abe3b41ba39c012761dbb2546b3e2ca0e7a6e96f82c358a657a5"} Sep 11 11:54:40.921779 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.921425 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" event={"ID":"df385b57-ee3a-49f9-a3ed-b201b1847f11","Type":"ContainerStarted","Data":"312f2ab05d19adcbac6d5fa6fad870c9ffba1fd6024021f9864cb6ebbe245fd2"} Sep 11 11:54:40.939318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:40.939270 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" podStartSLOduration=1.939256672 podStartE2EDuration="1.939256672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:54:40.936618965 +0000 UTC m=+1242.571212098" watchObservedRunningTime="2026-09-11 11:54:40.939256672 +0000 UTC m=+1242.573849805" Sep 11 11:54:45.799183 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:45.799144 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:54:50.232569 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:50.232482 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:50.232569 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:50.232543 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:50.244965 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:50.244946 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:54:50.990492 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:54:50.990465 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:55:00.041622 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:00.041535 2580 generic.go:356] "Generic (PLEG): container finished" podID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerID="26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384" exitCode=0 Sep 11 11:55:00.041622 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:00.041609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" event={"ID":"e20e2c8f-0764-480c-8842-2e3ed53a53fc","Type":"ContainerDied","Data":"26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384"} Sep 11 11:55:00.042106 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:00.042091 2580 scope.go:117] "RemoveContainer" containerID="26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384" Sep 11 11:55:00.043200 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:00.043174 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:55:00.043268 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:00.043242 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:55:00.043304 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:00.043273 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 11:55:01.049875 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:01.049838 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" event={"ID":"e20e2c8f-0764-480c-8842-2e3ed53a53fc","Type":"ContainerStarted","Data":"030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b"} Sep 11 11:55:01.532180 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:01.532146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:55:01.532374 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:01.532326 2580 patch_prober.go:28] interesting pod/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 11:55:01.532451 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:01.532380 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 11:55:02.532359 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:02.532320 2580 patch_prober.go:28] interesting pod/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 11:55:02.532763 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:02.532380 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 11:55:03.532330 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:03.532294 2580 patch_prober.go:28] interesting pod/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 11:55:03.532514 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:03.532350 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 11:55:04.532868 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:04.532826 2580 patch_prober.go:28] interesting pod/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 11:55:04.533274 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:04.532885 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 11:55:05.532698 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:05.532658 2580 patch_prober.go:28] interesting pod/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 11:55:05.532904 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:05.532726 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 11:55:06.536465 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:06.536437 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:55:06.536946 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:06.536816 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:55:06.537578 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:06.537560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:55:09.114479 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:09.114448 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:55:22.011312 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:22.011276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:55:39.090887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:39.090854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:55:54.242578 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:54.242543 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:55:59.047264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:55:59.047234 2580 scope.go:117] "RemoveContainer" containerID="fc47214afe9b1a51f61df230d65c2ca70936f4950598f8107d39b496f21eb7da" Sep 11 11:56:01.932108 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:01.932072 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh"] Sep 11 11:56:01.932585 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:01.932336 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" podUID="6439689a-bc56-45f3-acf4-5d32da43510d" containerName="maas-api" containerID="cri-o://5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab" gracePeriod=30 Sep 11 11:56:02.187773 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.187712 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:56:02.286009 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.285962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-metrics-certs\") pod \"6439689a-bc56-45f3-acf4-5d32da43510d\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " Sep 11 11:56:02.286281 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.286022 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6439689a-bc56-45f3-acf4-5d32da43510d-openshift-service-ca\") pod \"6439689a-bc56-45f3-acf4-5d32da43510d\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " Sep 11 11:56:02.286281 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.286129 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-maas-api-tls\") pod \"6439689a-bc56-45f3-acf4-5d32da43510d\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " Sep 11 11:56:02.286281 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.286184 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtv5q\" (UniqueName: \"kubernetes.io/projected/6439689a-bc56-45f3-acf4-5d32da43510d-kube-api-access-xtv5q\") pod \"6439689a-bc56-45f3-acf4-5d32da43510d\" (UID: \"6439689a-bc56-45f3-acf4-5d32da43510d\") " Sep 11 11:56:02.286507 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.286425 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6439689a-bc56-45f3-acf4-5d32da43510d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6439689a-bc56-45f3-acf4-5d32da43510d" (UID: "6439689a-bc56-45f3-acf4-5d32da43510d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:56:02.286590 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.286560 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6439689a-bc56-45f3-acf4-5d32da43510d-openshift-service-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:02.288600 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.288545 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6439689a-bc56-45f3-acf4-5d32da43510d" (UID: "6439689a-bc56-45f3-acf4-5d32da43510d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:56:02.288600 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.288576 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6439689a-bc56-45f3-acf4-5d32da43510d-kube-api-access-xtv5q" (OuterVolumeSpecName: "kube-api-access-xtv5q") pod "6439689a-bc56-45f3-acf4-5d32da43510d" (UID: "6439689a-bc56-45f3-acf4-5d32da43510d"). InnerVolumeSpecName "kube-api-access-xtv5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:56:02.288761 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.288632 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6439689a-bc56-45f3-acf4-5d32da43510d" (UID: "6439689a-bc56-45f3-acf4-5d32da43510d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:56:02.387069 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.387025 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-maas-api-tls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:02.387069 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.387069 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtv5q\" (UniqueName: \"kubernetes.io/projected/6439689a-bc56-45f3-acf4-5d32da43510d-kube-api-access-xtv5q\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:02.387264 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.387084 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6439689a-bc56-45f3-acf4-5d32da43510d-metrics-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:02.429777 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.429748 2580 generic.go:356] "Generic (PLEG): container finished" podID="6439689a-bc56-45f3-acf4-5d32da43510d" containerID="5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab" exitCode=0 Sep 11 11:56:02.429954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.429794 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" Sep 11 11:56:02.429954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.429828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" event={"ID":"6439689a-bc56-45f3-acf4-5d32da43510d","Type":"ContainerDied","Data":"5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab"} Sep 11 11:56:02.429954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.429877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh" event={"ID":"6439689a-bc56-45f3-acf4-5d32da43510d","Type":"ContainerDied","Data":"7c69b82049b1521b74a3f3ddf72afa3bce75ec0ede9a9b206e55529441f4708d"} Sep 11 11:56:02.429954 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.429899 2580 scope.go:117] "RemoveContainer" containerID="5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab" Sep 11 11:56:02.440022 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.440008 2580 scope.go:117] "RemoveContainer" containerID="5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab" Sep 11 11:56:02.440309 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:56:02.440289 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab\": container with ID starting with 5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab not found: ID does not exist" containerID="5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab" Sep 11 11:56:02.440381 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.440318 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab"} err="failed to get container status \"5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab\": rpc error: code = NotFound desc = could not find container \"5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab\": container with ID starting with 5092f05101044c1d014e5745f321d471f6d6d36d8881ac8388069f872b346eab not found: ID does not exist" Sep 11 11:56:02.508744 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.508711 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh"] Sep 11 11:56:02.521505 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.521478 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-307b7f39-7f8d7dbd65-vt2sh"] Sep 11 11:56:02.873255 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:02.873171 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6439689a-bc56-45f3-acf4-5d32da43510d" path="/var/lib/kubelet/pods/6439689a-bc56-45f3-acf4-5d32da43510d/volumes" Sep 11 11:56:16.491451 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:16.491415 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:56:28.209978 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:28.209899 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:56:45.499568 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:45.499529 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:56:53.884314 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:53.884279 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf"] Sep 11 11:56:53.884837 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:53.884479 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" containerID="cri-o://030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b" gracePeriod=30 Sep 11 11:56:59.148755 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.148736 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:56:59.231761 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231729 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-data\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.231761 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231769 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-podinfo\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.231967 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231819 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-certs\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.231967 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231846 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-token\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.231967 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231878 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-envoy\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.231967 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231912 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ml2t\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-kube-api-access-8ml2t\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.231967 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231939 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istiod-ca-cert\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.232254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.231998 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-credential-socket\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.232254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232071 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:56:59.232254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-socket\") pod \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\" (UID: \"e20e2c8f-0764-480c-8842-2e3ed53a53fc\") " Sep 11 11:56:59.232254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232189 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:56:59.232254 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232237 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:56:59.232482 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232334 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:56:59.232482 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232412 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:56:59.232574 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232557 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.232611 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232577 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.232611 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232594 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.232680 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232610 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.232680 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.232624 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.234247 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.234218 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:56:59.234419 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.234399 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-kube-api-access-8ml2t" (OuterVolumeSpecName: "kube-api-access-8ml2t") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "kube-api-access-8ml2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:56:59.234786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.234761 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 11:56:59.234786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.234781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "e20e2c8f-0764-480c-8842-2e3ed53a53fc" (UID: "e20e2c8f-0764-480c-8842-2e3ed53a53fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:56:59.333628 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.333593 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.333628 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.333622 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.333628 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.333631 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20e2c8f-0764-480c-8842-2e3ed53a53fc-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.333861 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.333640 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ml2t\" (UniqueName: \"kubernetes.io/projected/e20e2c8f-0764-480c-8842-2e3ed53a53fc-kube-api-access-8ml2t\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:56:59.772024 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.771985 2580 generic.go:356] "Generic (PLEG): container finished" podID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerID="030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b" exitCode=0 Sep 11 11:56:59.772024 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.772046 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" Sep 11 11:56:59.772287 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.772068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" event={"ID":"e20e2c8f-0764-480c-8842-2e3ed53a53fc","Type":"ContainerDied","Data":"030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b"} Sep 11 11:56:59.772287 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.772117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf" event={"ID":"e20e2c8f-0764-480c-8842-2e3ed53a53fc","Type":"ContainerDied","Data":"c6b90e684153794d72d0f19eff07f38412f975acec38e536fe5fbcfc029b60ee"} Sep 11 11:56:59.772287 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.772141 2580 scope.go:117] "RemoveContainer" containerID="030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b" Sep 11 11:56:59.782730 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.782712 2580 scope.go:117] "RemoveContainer" containerID="26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384" Sep 11 11:56:59.791105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.791086 2580 scope.go:117] "RemoveContainer" containerID="030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b" Sep 11 11:56:59.791469 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:56:59.791396 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b\": container with ID starting with 030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b not found: ID does not exist" containerID="030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b" Sep 11 11:56:59.791469 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.791432 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b"} err="failed to get container status \"030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b\": rpc error: code = NotFound desc = could not find container \"030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b\": container with ID starting with 030b8e0451eed97a38489898819edd6709f61227798e42a46108f65608ae9d4b not found: ID does not exist" Sep 11 11:56:59.791469 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.791458 2580 scope.go:117] "RemoveContainer" containerID="26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384" Sep 11 11:56:59.791762 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:56:59.791743 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384\": container with ID starting with 26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384 not found: ID does not exist" containerID="26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384" Sep 11 11:56:59.791807 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.791769 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384"} err="failed to get container status \"26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384\": rpc error: code = NotFound desc = could not find container \"26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384\": container with ID starting with 26a4329fc35cfd9c39bd9da2990e9cc3af347f8d9b1b614c6553b1632a226384 not found: ID does not exist" Sep 11 11:56:59.794424 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.794403 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf"] Sep 11 11:56:59.797360 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:56:59.797343 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-307b7f39-openshift-default-c8cffb556-dlsnf"] Sep 11 11:57:00.873219 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:00.873183 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" path="/var/lib/kubelet/pods/e20e2c8f-0764-480c-8842-2e3ed53a53fc/volumes" Sep 11 11:57:01.293668 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:01.293633 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:57:14.499551 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:14.499514 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:57:25.588557 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.588516 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8014-28af35"] Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.588900 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6439689a-bc56-45f3-acf4-5d32da43510d" containerName="maas-api" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.588911 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6439689a-bc56-45f3-acf4-5d32da43510d" containerName="maas-api" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.588930 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.588936 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.588950 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.588955 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.589016 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6439689a-bc56-45f3-acf4-5d32da43510d" containerName="maas-api" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.589023 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" Sep 11 11:57:25.589023 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.589045 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20e2c8f-0764-480c-8842-2e3ed53a53fc" containerName="istio-proxy" Sep 11 11:57:25.593631 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.593610 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:25.596086 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.596062 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5xz2c\"" Sep 11 11:57:25.598353 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.598327 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-28af35"] Sep 11 11:57:25.683445 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.683417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkv8q\" (UniqueName: \"kubernetes.io/projected/33ff3f48-7256-42bf-98e6-082e32c921db-kube-api-access-dkv8q\") pod \"test-curl-8014-28af35\" (UID: \"33ff3f48-7256-42bf-98e6-082e32c921db\") " pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:25.784194 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.784153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkv8q\" (UniqueName: \"kubernetes.io/projected/33ff3f48-7256-42bf-98e6-082e32c921db-kube-api-access-dkv8q\") pod \"test-curl-8014-28af35\" (UID: \"33ff3f48-7256-42bf-98e6-082e32c921db\") " pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:25.792964 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.792935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkv8q\" (UniqueName: \"kubernetes.io/projected/33ff3f48-7256-42bf-98e6-082e32c921db-kube-api-access-dkv8q\") pod \"test-curl-8014-28af35\" (UID: \"33ff3f48-7256-42bf-98e6-082e32c921db\") " pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:25.905273 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:25.905248 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:26.193054 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:26.192938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-28af35"] Sep 11 11:57:26.198423 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:57:26.198398 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33ff3f48_7256_42bf_98e6_082e32c921db.slice/crio-1dab1c3dc8231779d5822a20119525ea37f2b9efe7055d15891e62b415092ffe WatchSource:0}: Error finding container 1dab1c3dc8231779d5822a20119525ea37f2b9efe7055d15891e62b415092ffe: Status 404 returned error can't find the container with id 1dab1c3dc8231779d5822a20119525ea37f2b9efe7055d15891e62b415092ffe Sep 11 11:57:26.941436 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:26.941406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-28af35" event={"ID":"33ff3f48-7256-42bf-98e6-082e32c921db","Type":"ContainerStarted","Data":"d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0"} Sep 11 11:57:26.941436 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:26.941451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-28af35" event={"ID":"33ff3f48-7256-42bf-98e6-082e32c921db","Type":"ContainerStarted","Data":"1dab1c3dc8231779d5822a20119525ea37f2b9efe7055d15891e62b415092ffe"} Sep 11 11:57:26.959056 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:26.958989 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8014-28af35" podStartSLOduration=1.958974815 podStartE2EDuration="1.958974815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:57:26.956605555 +0000 UTC m=+1408.591198689" watchObservedRunningTime="2026-09-11 11:57:26.958974815 +0000 UTC m=+1408.593567947" Sep 11 11:57:27.531891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:27.531860 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8014-28af35"] Sep 11 11:57:27.535315 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:27.535289 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8014-28af35"] Sep 11 11:57:27.946707 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:57:27.946681 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8014-28af35" container="test-curl-8014-28af35" Sep 11 11:57:28.195105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:28.195063 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:57:28.816722 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:28.816682 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2"] Sep 11 11:57:28.952448 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:28.952385 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8014-28af35" podUID="33ff3f48-7256-42bf-98e6-082e32c921db" containerName="test-curl-8014-28af35" containerID="cri-o://d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0" gracePeriod=2 Sep 11 11:57:31.139429 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.139400 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:31.234659 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.234576 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkv8q\" (UniqueName: \"kubernetes.io/projected/33ff3f48-7256-42bf-98e6-082e32c921db-kube-api-access-dkv8q\") pod \"33ff3f48-7256-42bf-98e6-082e32c921db\" (UID: \"33ff3f48-7256-42bf-98e6-082e32c921db\") " Sep 11 11:57:31.236938 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.236897 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ff3f48-7256-42bf-98e6-082e32c921db-kube-api-access-dkv8q" (OuterVolumeSpecName: "kube-api-access-dkv8q") pod "33ff3f48-7256-42bf-98e6-082e32c921db" (UID: "33ff3f48-7256-42bf-98e6-082e32c921db"). InnerVolumeSpecName "kube-api-access-dkv8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:57:31.336169 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.336128 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkv8q\" (UniqueName: \"kubernetes.io/projected/33ff3f48-7256-42bf-98e6-082e32c921db-kube-api-access-dkv8q\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:57:31.970835 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.970805 2580 generic.go:356] "Generic (PLEG): container finished" podID="33ff3f48-7256-42bf-98e6-082e32c921db" containerID="d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0" exitCode=137 Sep 11 11:57:31.971027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.970891 2580 scope.go:117] "RemoveContainer" containerID="d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0" Sep 11 11:57:31.971027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.970915 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:31.981190 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:57:31.981164 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-28af35\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="33ff3f48-7256-42bf-98e6-082e32c921db" pod="opendatahub/test-curl-8014-28af35" Sep 11 11:57:31.981190 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.981179 2580 scope.go:117] "RemoveContainer" containerID="d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0" Sep 11 11:57:31.981541 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:57:31.981511 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0\": container with ID starting with d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0 not found: ID does not exist" containerID="d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0" Sep 11 11:57:31.981636 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:31.981540 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0"} err="failed to get container status \"d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0\": rpc error: code = NotFound desc = could not find container \"d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0\": container with ID starting with d353a290e4894e8447bb3f5e9966c34f1ec8a917389f6bcb4e9dabef4f8f22e0 not found: ID does not exist" Sep 11 11:57:32.873362 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:32.873328 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ff3f48-7256-42bf-98e6-082e32c921db" path="/var/lib/kubelet/pods/33ff3f48-7256-42bf-98e6-082e32c921db/volumes" Sep 11 11:57:38.208027 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:38.207986 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:57:42.221609 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.221576 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb"] Sep 11 11:57:42.222013 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.221999 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ff3f48-7256-42bf-98e6-082e32c921db" containerName="test-curl-8014-28af35" Sep 11 11:57:42.222103 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.222016 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ff3f48-7256-42bf-98e6-082e32c921db" containerName="test-curl-8014-28af35" Sep 11 11:57:42.222141 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.222112 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ff3f48-7256-42bf-98e6-082e32c921db" containerName="test-curl-8014-28af35" Sep 11 11:57:42.226534 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.226517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.228790 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.228767 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-272c5e\"/\"sub-test-model-272c5e-kserve-self-signed-certs\"" Sep 11 11:57:42.235162 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.235131 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb"] Sep 11 11:57:42.238237 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.238212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tls-certs\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.238366 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.238260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tmp-dir\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.238366 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.238308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-model-cache\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.238366 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.238333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-home\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.238510 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.238436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpgt\" (UniqueName: \"kubernetes.io/projected/3209ad4a-2d34-4975-8b35-1dce052a2ddd-kube-api-access-rzpgt\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.238570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.238506 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-dshm\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.339814 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.339786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-dshm\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.339998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.339854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tls-certs\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.339998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.339880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tmp-dir\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.339998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.339900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-model-cache\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.339998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.339914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-home\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.339998 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.339967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpgt\" (UniqueName: \"kubernetes.io/projected/3209ad4a-2d34-4975-8b35-1dce052a2ddd-kube-api-access-rzpgt\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.340330 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.340304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tmp-dir\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.340395 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.340356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-model-cache\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.340395 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.340374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-home\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.342294 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.342271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-dshm\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.342763 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.342743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tls-certs\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.347574 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.347546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpgt\" (UniqueName: \"kubernetes.io/projected/3209ad4a-2d34-4975-8b35-1dce052a2ddd-kube-api-access-rzpgt\") pod \"sub-test-model-272c5e-kserve-5db7448df9-jnjpb\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.537342 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.537263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:42.812131 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:42.812104 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb"] Sep 11 11:57:43.034776 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:43.034743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" event={"ID":"3209ad4a-2d34-4975-8b35-1dce052a2ddd","Type":"ContainerStarted","Data":"79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e"} Sep 11 11:57:43.034776 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:43.034787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" event={"ID":"3209ad4a-2d34-4975-8b35-1dce052a2ddd","Type":"ContainerStarted","Data":"72bd3a872a43ade258aa1897c40e5d6a5f1179bb28e1dbe736d622e460358d63"} Sep 11 11:57:43.052771 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:43.052727 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" podStartSLOduration=1.052713553 podStartE2EDuration="1.052713553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:57:43.049802259 +0000 UTC m=+1424.684395392" watchObservedRunningTime="2026-09-11 11:57:43.052713553 +0000 UTC m=+1424.687306685" Sep 11 11:57:43.836413 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:43.836368 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" podUID="df385b57-ee3a-49f9-a3ed-b201b1847f11" containerName="main" containerID="cri-o://6f47d00b11b0abe3b41ba39c012761dbb2546b3e2ca0e7a6e96f82c358a657a5" gracePeriod=45 Sep 11 11:57:44.042516 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.042485 2580 generic.go:356] "Generic (PLEG): container finished" podID="df385b57-ee3a-49f9-a3ed-b201b1847f11" containerID="6f47d00b11b0abe3b41ba39c012761dbb2546b3e2ca0e7a6e96f82c358a657a5" exitCode=0 Sep 11 11:57:44.042678 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.042558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" event={"ID":"df385b57-ee3a-49f9-a3ed-b201b1847f11","Type":"ContainerDied","Data":"6f47d00b11b0abe3b41ba39c012761dbb2546b3e2ca0e7a6e96f82c358a657a5"} Sep 11 11:57:44.098418 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.098358 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:57:44.157440 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157413 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-home\") pod \"df385b57-ee3a-49f9-a3ed-b201b1847f11\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " Sep 11 11:57:44.157440 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157441 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-model-cache\") pod \"df385b57-ee3a-49f9-a3ed-b201b1847f11\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " Sep 11 11:57:44.157640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157467 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xs7k\" (UniqueName: \"kubernetes.io/projected/df385b57-ee3a-49f9-a3ed-b201b1847f11-kube-api-access-4xs7k\") pod \"df385b57-ee3a-49f9-a3ed-b201b1847f11\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " Sep 11 11:57:44.157640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157520 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-dshm\") pod \"df385b57-ee3a-49f9-a3ed-b201b1847f11\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " Sep 11 11:57:44.157640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157559 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df385b57-ee3a-49f9-a3ed-b201b1847f11-tls-certs\") pod \"df385b57-ee3a-49f9-a3ed-b201b1847f11\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " Sep 11 11:57:44.157640 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157591 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-tmp-dir\") pod \"df385b57-ee3a-49f9-a3ed-b201b1847f11\" (UID: \"df385b57-ee3a-49f9-a3ed-b201b1847f11\") " Sep 11 11:57:44.157854 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157703 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-model-cache" (OuterVolumeSpecName: "model-cache") pod "df385b57-ee3a-49f9-a3ed-b201b1847f11" (UID: "df385b57-ee3a-49f9-a3ed-b201b1847f11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:57:44.157854 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-home" (OuterVolumeSpecName: "home") pod "df385b57-ee3a-49f9-a3ed-b201b1847f11" (UID: "df385b57-ee3a-49f9-a3ed-b201b1847f11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:57:44.157962 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157904 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df385b57-ee3a-49f9-a3ed-b201b1847f11" (UID: "df385b57-ee3a-49f9-a3ed-b201b1847f11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:57:44.158005 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157971 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:57:44.158005 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.157996 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:57:44.159857 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.159834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-dshm" (OuterVolumeSpecName: "dshm") pod "df385b57-ee3a-49f9-a3ed-b201b1847f11" (UID: "df385b57-ee3a-49f9-a3ed-b201b1847f11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:57:44.159930 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.159909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df385b57-ee3a-49f9-a3ed-b201b1847f11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df385b57-ee3a-49f9-a3ed-b201b1847f11" (UID: "df385b57-ee3a-49f9-a3ed-b201b1847f11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:57:44.159975 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.159958 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df385b57-ee3a-49f9-a3ed-b201b1847f11-kube-api-access-4xs7k" (OuterVolumeSpecName: "kube-api-access-4xs7k") pod "df385b57-ee3a-49f9-a3ed-b201b1847f11" (UID: "df385b57-ee3a-49f9-a3ed-b201b1847f11"). InnerVolumeSpecName "kube-api-access-4xs7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:57:44.259500 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.259467 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:57:44.259500 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.259496 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df385b57-ee3a-49f9-a3ed-b201b1847f11-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:57:44.259500 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.259508 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df385b57-ee3a-49f9-a3ed-b201b1847f11-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:57:44.259716 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:44.259519 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xs7k\" (UniqueName: \"kubernetes.io/projected/df385b57-ee3a-49f9-a3ed-b201b1847f11-kube-api-access-4xs7k\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:57:45.050321 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:45.050280 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" event={"ID":"df385b57-ee3a-49f9-a3ed-b201b1847f11","Type":"ContainerDied","Data":"312f2ab05d19adcbac6d5fa6fad870c9ffba1fd6024021f9864cb6ebbe245fd2"} Sep 11 11:57:45.050728 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:45.050330 2580 scope.go:117] "RemoveContainer" containerID="6f47d00b11b0abe3b41ba39c012761dbb2546b3e2ca0e7a6e96f82c358a657a5" Sep 11 11:57:45.050728 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:45.050290 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2" Sep 11 11:57:45.069379 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:45.069348 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2"] Sep 11 11:57:45.071661 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:45.071639 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/auth-test-model-272c5e-kserve-6577f549f5-wxss2"] Sep 11 11:57:45.498325 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:45.498292 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:57:46.873241 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:46.873206 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df385b57-ee3a-49f9-a3ed-b201b1847f11" path="/var/lib/kubelet/pods/df385b57-ee3a-49f9-a3ed-b201b1847f11/volumes" Sep 11 11:57:52.538079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.537970 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:52.538079 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.538066 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:52.551709 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.551684 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:52.552416 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.552396 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f"] Sep 11 11:57:52.552785 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.552772 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df385b57-ee3a-49f9-a3ed-b201b1847f11" containerName="main" Sep 11 11:57:52.552831 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.552786 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="df385b57-ee3a-49f9-a3ed-b201b1847f11" containerName="main" Sep 11 11:57:52.552884 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.552874 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="df385b57-ee3a-49f9-a3ed-b201b1847f11" containerName="main" Sep 11 11:57:52.557409 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.557392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.560551 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.560522 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f9def1-serving-cert\"" Sep 11 11:57:52.566803 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.566780 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f"] Sep 11 11:57:52.637318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.637282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.637318 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.637321 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-maas-api-tls\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.637514 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.637344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zzfp\" (UniqueName: \"kubernetes.io/projected/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-kube-api-access-4zzfp\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.637514 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.637476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-metrics-certs\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.738419 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.738385 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-metrics-certs\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.738620 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.738462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.738620 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.738485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-maas-api-tls\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.738620 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.738507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zzfp\" (UniqueName: \"kubernetes.io/projected/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-kube-api-access-4zzfp\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.739323 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.739232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.741075 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.741054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-metrics-certs\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.741222 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.741203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-maas-api-tls\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.747472 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.747448 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zzfp\" (UniqueName: \"kubernetes.io/projected/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-kube-api-access-4zzfp\") pod \"maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:52.869401 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:52.869317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:53.109147 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:53.109112 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:57:53.150572 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:53.150523 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f"] Sep 11 11:57:53.151231 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:57:53.151204 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a23fa5_6d4a_4ae4_b40a_c2a8745ff821.slice/crio-43a85eadd2b9b1078b7120f1b0975a60cc4b81a4628485b3d69fddf8f4d2464a WatchSource:0}: Error finding container 43a85eadd2b9b1078b7120f1b0975a60cc4b81a4628485b3d69fddf8f4d2464a: Status 404 returned error can't find the container with id 43a85eadd2b9b1078b7120f1b0975a60cc4b81a4628485b3d69fddf8f4d2464a Sep 11 11:57:54.105105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:54.105062 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" event={"ID":"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821","Type":"ContainerStarted","Data":"39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2"} Sep 11 11:57:54.105575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:54.105171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" event={"ID":"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821","Type":"ContainerStarted","Data":"43a85eadd2b9b1078b7120f1b0975a60cc4b81a4628485b3d69fddf8f4d2464a"} Sep 11 11:57:54.105575 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:54.105458 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:57:54.125957 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:54.125915 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" podStartSLOduration=1.736725606 podStartE2EDuration="2.125901377s" totalImagesPullingTime="389.175771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:57:54.12424398 +0000 UTC m=+1435.758837113" watchObservedRunningTime="2026-09-11 11:57:54.125901377 +0000 UTC m=+1435.760494509" Sep 11 11:57:58.998646 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:57:58.998614 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:58:00.116140 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.116113 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:58:00.121981 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.121946 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm"] Sep 11 11:58:00.126459 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.126442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.128905 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.128884 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-117711\"/\"openshift-service-ca.crt\"" Sep 11 11:58:00.129022 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.128895 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-117711\"/\"kube-root-ca.crt\"" Sep 11 11:58:00.129022 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.128919 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-117711\"/\"default-dockercfg-ndglz\"" Sep 11 11:58:00.129022 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.128925 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-117711\"/\"sub-test-model-117711-kserve-self-signed-certs\"" Sep 11 11:58:00.135290 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.135266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm"] Sep 11 11:58:00.212042 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.212015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-dshm\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.212211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.212103 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-home\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.212211 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.212136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-model-cache\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.212332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.212302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bda71c-b34d-41ee-863a-f53dc8f7e229-tls-certs\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.212397 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.212380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bldl2\" (UniqueName: \"kubernetes.io/projected/06bda71c-b34d-41ee-863a-f53dc8f7e229-kube-api-access-bldl2\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.212435 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.212407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-tmp-dir\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313331 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-dshm\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-home\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-model-cache\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bda71c-b34d-41ee-863a-f53dc8f7e229-tls-certs\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313512 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bldl2\" (UniqueName: \"kubernetes.io/projected/06bda71c-b34d-41ee-863a-f53dc8f7e229-kube-api-access-bldl2\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313729 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-tmp-dir\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313857 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-model-cache\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313922 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-home\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.313922 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.313886 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-tmp-dir\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.315695 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.315675 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-dshm\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.315994 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.315977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bda71c-b34d-41ee-863a-f53dc8f7e229-tls-certs\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.322123 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.322102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bldl2\" (UniqueName: \"kubernetes.io/projected/06bda71c-b34d-41ee-863a-f53dc8f7e229-kube-api-access-bldl2\") pod \"sub-test-model-117711-kserve-98f4f8d7f-mzzpm\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.437819 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.437792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:00.719503 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:00.719420 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm"] Sep 11 11:58:00.721869 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:58:00.721835 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06bda71c_b34d_41ee_863a_f53dc8f7e229.slice/crio-3a2f6f18db82adb8e632656f4b62779cfb9b2b11f8dd5c4c68e1d95887c17e28 WatchSource:0}: Error finding container 3a2f6f18db82adb8e632656f4b62779cfb9b2b11f8dd5c4c68e1d95887c17e28: Status 404 returned error can't find the container with id 3a2f6f18db82adb8e632656f4b62779cfb9b2b11f8dd5c4c68e1d95887c17e28 Sep 11 11:58:01.152408 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:01.152373 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" event={"ID":"06bda71c-b34d-41ee-863a-f53dc8f7e229","Type":"ContainerStarted","Data":"566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a"} Sep 11 11:58:01.152786 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:01.152420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" event={"ID":"06bda71c-b34d-41ee-863a-f53dc8f7e229","Type":"ContainerStarted","Data":"3a2f6f18db82adb8e632656f4b62779cfb9b2b11f8dd5c4c68e1d95887c17e28"} Sep 11 11:58:01.171580 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:01.171536 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" podStartSLOduration=1.17152152 podStartE2EDuration="1.17152152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:01.168299423 +0000 UTC m=+1442.802892558" watchObservedRunningTime="2026-09-11 11:58:01.17152152 +0000 UTC m=+1442.806114653" Sep 11 11:58:10.438228 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:10.438194 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:10.438709 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:10.438243 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:10.450580 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:10.450557 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:11.221676 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:11.221651 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:58:16.099236 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:16.099201 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:58:21.657616 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.657583 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9"] Sep 11 11:58:21.663410 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.663388 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.665673 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.665648 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-641847-serving-cert\"" Sep 11 11:58:21.673339 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.673305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9"] Sep 11 11:58:21.819376 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.819340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jdg\" (UniqueName: \"kubernetes.io/projected/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-kube-api-access-d7jdg\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.819376 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.819377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-maas-api-tls\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.819571 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.819410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-metrics-certs\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.819571 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.819471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.921081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.920977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jdg\" (UniqueName: \"kubernetes.io/projected/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-kube-api-access-d7jdg\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.921081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.921008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-maas-api-tls\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.921081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.921062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-metrics-certs\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.921361 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.921108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.921780 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.921758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.923621 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.923596 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-metrics-certs\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.923713 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.923624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-maas-api-tls\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.930159 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.930138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jdg\" (UniqueName: \"kubernetes.io/projected/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-kube-api-access-d7jdg\") pod \"maas-api-e2e-api-b-641847-f954d6464-wlgl9\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:21.978088 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:21.978058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:22.254130 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:22.254104 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9"] Sep 11 11:58:22.255711 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:58:22.255684 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdb5162_ff3a_40e7_a47d_b4ce7280f7bc.slice/crio-6f7f061ac112ee9c9c5baf5cd00bee6a5dde3a56892e5ae95e856def76acffed WatchSource:0}: Error finding container 6f7f061ac112ee9c9c5baf5cd00bee6a5dde3a56892e5ae95e856def76acffed: Status 404 returned error can't find the container with id 6f7f061ac112ee9c9c5baf5cd00bee6a5dde3a56892e5ae95e856def76acffed Sep 11 11:58:22.280732 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:22.280698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" event={"ID":"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc","Type":"ContainerStarted","Data":"6f7f061ac112ee9c9c5baf5cd00bee6a5dde3a56892e5ae95e856def76acffed"} Sep 11 11:58:23.288816 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:23.288782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" event={"ID":"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc","Type":"ContainerStarted","Data":"bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb"} Sep 11 11:58:23.289245 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:23.288874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:23.306618 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:23.306574 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" podStartSLOduration=1.826495112 podStartE2EDuration="2.306559829s" totalImagesPullingTime="480.064717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:23.305314037 +0000 UTC m=+1464.939907180" watchObservedRunningTime="2026-09-11 11:58:23.306559829 +0000 UTC m=+1464.941152962" Sep 11 11:58:29.300584 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:29.300555 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:58:32.408017 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:32.407976 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:58:50.993742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:50.993708 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:58:53.951933 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:53.951901 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8014-f0410e"] Sep 11 11:58:53.956604 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:53.956586 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:58:53.959523 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:53.959504 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5xz2c\"" Sep 11 11:58:53.962651 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:53.962627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-f0410e"] Sep 11 11:58:54.134006 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:54.133969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtt6\" (UniqueName: \"kubernetes.io/projected/b0342280-ceec-4b81-b399-e7571c1fbf67-kube-api-access-2xtt6\") pod \"test-curl-8014-f0410e\" (UID: \"b0342280-ceec-4b81-b399-e7571c1fbf67\") " pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:58:54.234602 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:54.234513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtt6\" (UniqueName: \"kubernetes.io/projected/b0342280-ceec-4b81-b399-e7571c1fbf67-kube-api-access-2xtt6\") pod \"test-curl-8014-f0410e\" (UID: \"b0342280-ceec-4b81-b399-e7571c1fbf67\") " pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:58:54.243191 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:54.243168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtt6\" (UniqueName: \"kubernetes.io/projected/b0342280-ceec-4b81-b399-e7571c1fbf67-kube-api-access-2xtt6\") pod \"test-curl-8014-f0410e\" (UID: \"b0342280-ceec-4b81-b399-e7571c1fbf67\") " pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:58:54.268247 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:54.268225 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:58:54.562923 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:54.562899 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-f0410e"] Sep 11 11:58:54.564531 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:58:54.564498 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0342280_ceec_4b81_b399_e7571c1fbf67.slice/crio-f94f6d6f0d524bad15f7859ba8bf6a4f87d7f02b0dd74920d56f53f1c6d942ed WatchSource:0}: Error finding container f94f6d6f0d524bad15f7859ba8bf6a4f87d7f02b0dd74920d56f53f1c6d942ed: Status 404 returned error can't find the container with id f94f6d6f0d524bad15f7859ba8bf6a4f87d7f02b0dd74920d56f53f1c6d942ed Sep 11 11:58:55.477091 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:55.477054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-f0410e" event={"ID":"b0342280-ceec-4b81-b399-e7571c1fbf67","Type":"ContainerStarted","Data":"373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee"} Sep 11 11:58:55.477091 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:55.477098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-f0410e" event={"ID":"b0342280-ceec-4b81-b399-e7571c1fbf67","Type":"ContainerStarted","Data":"f94f6d6f0d524bad15f7859ba8bf6a4f87d7f02b0dd74920d56f53f1c6d942ed"} Sep 11 11:58:55.491892 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:55.491849 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8014-f0410e" podStartSLOduration=2.491835994 podStartE2EDuration="2.491835994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:55.490174567 +0000 UTC m=+1497.124767710" watchObservedRunningTime="2026-09-11 11:58:55.491835994 +0000 UTC m=+1497.126429127" Sep 11 11:58:56.045472 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.045430 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8014-f0410e"] Sep 11 11:58:56.054279 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.054252 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8014-f0410e"] Sep 11 11:58:56.165344 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.165310 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8014-cf42ef"] Sep 11 11:58:56.165776 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.165763 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0342280-ceec-4b81-b399-e7571c1fbf67" containerName="test-curl-8014-f0410e" Sep 11 11:58:56.165829 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.165778 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0342280-ceec-4b81-b399-e7571c1fbf67" containerName="test-curl-8014-f0410e" Sep 11 11:58:56.165868 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.165844 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0342280-ceec-4b81-b399-e7571c1fbf67" containerName="test-curl-8014-f0410e" Sep 11 11:58:56.168869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.168855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:58:56.176119 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.176091 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-cf42ef"] Sep 11 11:58:56.254362 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.254319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvr7\" (UniqueName: \"kubernetes.io/projected/7ef01991-b2f6-4f6c-b265-b55bee4f3438-kube-api-access-6xvr7\") pod \"test-curl-8014-cf42ef\" (UID: \"7ef01991-b2f6-4f6c-b265-b55bee4f3438\") " pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:58:56.355222 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.355190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvr7\" (UniqueName: \"kubernetes.io/projected/7ef01991-b2f6-4f6c-b265-b55bee4f3438-kube-api-access-6xvr7\") pod \"test-curl-8014-cf42ef\" (UID: \"7ef01991-b2f6-4f6c-b265-b55bee4f3438\") " pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:58:56.364791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.364769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvr7\" (UniqueName: \"kubernetes.io/projected/7ef01991-b2f6-4f6c-b265-b55bee4f3438-kube-api-access-6xvr7\") pod \"test-curl-8014-cf42ef\" (UID: \"7ef01991-b2f6-4f6c-b265-b55bee4f3438\") " pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:58:56.479869 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.479781 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:58:56.482737 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:58:56.482634 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8014-f0410e" container="test-curl-8014-f0410e" Sep 11 11:58:56.755099 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:56.755069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-cf42ef"] Sep 11 11:58:56.755706 ip-10-0-135-183 kubenswrapper[2580]: W0911 11:58:56.755679 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ef01991_b2f6_4f6c_b265_b55bee4f3438.slice/crio-e19722db940350979f49883e89a5c7e207436a9540c48117e1a5c3992e828f10 WatchSource:0}: Error finding container e19722db940350979f49883e89a5c7e207436a9540c48117e1a5c3992e828f10: Status 404 returned error can't find the container with id e19722db940350979f49883e89a5c7e207436a9540c48117e1a5c3992e828f10 Sep 11 11:58:57.489497 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:57.489461 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-cf42ef" event={"ID":"7ef01991-b2f6-4f6c-b265-b55bee4f3438","Type":"ContainerStarted","Data":"004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db"} Sep 11 11:58:57.489962 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:57.489511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-cf42ef" event={"ID":"7ef01991-b2f6-4f6c-b265-b55bee4f3438","Type":"ContainerStarted","Data":"e19722db940350979f49883e89a5c7e207436a9540c48117e1a5c3992e828f10"} Sep 11 11:58:57.489962 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:57.489533 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8014-f0410e" podUID="b0342280-ceec-4b81-b399-e7571c1fbf67" containerName="test-curl-8014-f0410e" containerID="cri-o://373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee" gracePeriod=2 Sep 11 11:58:57.505583 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:57.505539 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8014-cf42ef" podStartSLOduration=1.505522799 podStartE2EDuration="1.505522799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:58:57.502956233 +0000 UTC m=+1499.137549366" watchObservedRunningTime="2026-09-11 11:58:57.505522799 +0000 UTC m=+1499.140115931" Sep 11 11:58:58.069226 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:58.069192 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8014-cf42ef"] Sep 11 11:58:58.074517 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:58.074486 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8014-cf42ef"] Sep 11 11:58:58.495776 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:58:58.495746 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8014-cf42ef" container="test-curl-8014-cf42ef" Sep 11 11:58:58.972654 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:58.972626 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:58:58.974175 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:58.974156 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 11:58:59.501012 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:59.500977 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8014-cf42ef" podUID="7ef01991-b2f6-4f6c-b265-b55bee4f3438" containerName="test-curl-8014-cf42ef" containerID="cri-o://004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db" gracePeriod=2 Sep 11 11:58:59.661332 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:59.661308 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:58:59.791816 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:59.791736 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xtt6\" (UniqueName: \"kubernetes.io/projected/b0342280-ceec-4b81-b399-e7571c1fbf67-kube-api-access-2xtt6\") pod \"b0342280-ceec-4b81-b399-e7571c1fbf67\" (UID: \"b0342280-ceec-4b81-b399-e7571c1fbf67\") " Sep 11 11:58:59.793891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:59.793868 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0342280-ceec-4b81-b399-e7571c1fbf67-kube-api-access-2xtt6" (OuterVolumeSpecName: "kube-api-access-2xtt6") pod "b0342280-ceec-4b81-b399-e7571c1fbf67" (UID: "b0342280-ceec-4b81-b399-e7571c1fbf67"). InnerVolumeSpecName "kube-api-access-2xtt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:58:59.893344 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:58:59.893300 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtt6\" (UniqueName: \"kubernetes.io/projected/b0342280-ceec-4b81-b399-e7571c1fbf67-kube-api-access-2xtt6\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:00.144393 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:00.144307 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb"] Sep 11 11:59:00.507667 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:00.507636 2580 generic.go:356] "Generic (PLEG): container finished" podID="b0342280-ceec-4b81-b399-e7571c1fbf67" containerID="373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee" exitCode=137 Sep 11 11:59:00.508061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:00.507675 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:59:00.508061 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:00.507730 2580 scope.go:117] "RemoveContainer" containerID="373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee" Sep 11 11:59:00.517266 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:00.517242 2580 scope.go:117] "RemoveContainer" containerID="373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee" Sep 11 11:59:00.517513 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:00.517497 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee\": container with ID starting with 373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee not found: ID does not exist" containerID="373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee" Sep 11 11:59:00.517555 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:00.517521 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee"} err="failed to get container status \"373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee\": rpc error: code = NotFound desc = could not find container \"373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee\": container with ID starting with 373737f7c98f4d72dd23bc18b9dc40260ca1d01840507b2b3b36f8f11f7d59ee not found: ID does not exist" Sep 11 11:59:00.518025 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:00.518002 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-f0410e\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="b0342280-ceec-4b81-b399-e7571c1fbf67" pod="opendatahub/test-curl-8014-f0410e" Sep 11 11:59:00.873583 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:00.873505 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0342280-ceec-4b81-b399-e7571c1fbf67" path="/var/lib/kubelet/pods/b0342280-ceec-4b81-b399-e7571c1fbf67/volumes" Sep 11 11:59:01.116299 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:01.116270 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm"] Sep 11 11:59:01.695253 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:01.695231 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:59:01.815960 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:01.815868 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xvr7\" (UniqueName: \"kubernetes.io/projected/7ef01991-b2f6-4f6c-b265-b55bee4f3438-kube-api-access-6xvr7\") pod \"7ef01991-b2f6-4f6c-b265-b55bee4f3438\" (UID: \"7ef01991-b2f6-4f6c-b265-b55bee4f3438\") " Sep 11 11:59:01.817975 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:01.817952 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef01991-b2f6-4f6c-b265-b55bee4f3438-kube-api-access-6xvr7" (OuterVolumeSpecName: "kube-api-access-6xvr7") pod "7ef01991-b2f6-4f6c-b265-b55bee4f3438" (UID: "7ef01991-b2f6-4f6c-b265-b55bee4f3438"). InnerVolumeSpecName "kube-api-access-6xvr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:59:01.917152 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:01.917119 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xvr7\" (UniqueName: \"kubernetes.io/projected/7ef01991-b2f6-4f6c-b265-b55bee4f3438-kube-api-access-6xvr7\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:02.522673 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:02.522638 2580 generic.go:356] "Generic (PLEG): container finished" podID="7ef01991-b2f6-4f6c-b265-b55bee4f3438" containerID="004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db" exitCode=137 Sep 11 11:59:02.522843 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:02.522691 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:59:02.522843 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:02.522735 2580 scope.go:117] "RemoveContainer" containerID="004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db" Sep 11 11:59:02.532333 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:02.532253 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8014-cf42ef\" is forbidden: User \"system:node:ip-10-0-135-183.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-183.ec2.internal' and this object" podUID="7ef01991-b2f6-4f6c-b265-b55bee4f3438" pod="opendatahub/test-curl-8014-cf42ef" Sep 11 11:59:02.532408 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:02.532340 2580 scope.go:117] "RemoveContainer" containerID="004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db" Sep 11 11:59:02.532632 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:02.532612 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db\": container with ID starting with 004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db not found: ID does not exist" containerID="004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db" Sep 11 11:59:02.532688 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:02.532642 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db"} err="failed to get container status \"004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db\": rpc error: code = NotFound desc = could not find container \"004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db\": container with ID starting with 004095795bbbce90d3949a0a6ba8f7e3350413c8354b186b4ca1806510eaf9db not found: ID does not exist" Sep 11 11:59:02.875440 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:02.875358 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef01991-b2f6-4f6c-b265-b55bee4f3438" path="/var/lib/kubelet/pods/7ef01991-b2f6-4f6c-b265-b55bee4f3438/volumes" Sep 11 11:59:03.928902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:03.928869 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f"] Sep 11 11:59:03.929519 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:03.929151 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" podUID="d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" containerName="maas-api" containerID="cri-o://39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2" gracePeriod=30 Sep 11 11:59:04.186885 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.186819 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:59:04.234213 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.234186 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-maas-api-tls\") pod \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " Sep 11 11:59:04.234375 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.234225 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-openshift-service-ca\") pod \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " Sep 11 11:59:04.234375 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.234307 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zzfp\" (UniqueName: \"kubernetes.io/projected/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-kube-api-access-4zzfp\") pod \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " Sep 11 11:59:04.234375 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.234345 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-metrics-certs\") pod \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\" (UID: \"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821\") " Sep 11 11:59:04.234717 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.234688 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" (UID: "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:59:04.234828 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.234803 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-openshift-service-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:04.236902 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.236870 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" (UID: "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:59:04.237060 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.237011 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" (UID: "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:59:04.237120 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.237059 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-kube-api-access-4zzfp" (OuterVolumeSpecName: "kube-api-access-4zzfp") pod "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" (UID: "d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821"). InnerVolumeSpecName "kube-api-access-4zzfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:59:04.335423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.335385 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zzfp\" (UniqueName: \"kubernetes.io/projected/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-kube-api-access-4zzfp\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:04.335423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.335413 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-metrics-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:04.335423 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.335422 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821-maas-api-tls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:04.536531 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.536428 2580 generic.go:356] "Generic (PLEG): container finished" podID="d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" containerID="39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2" exitCode=0 Sep 11 11:59:04.536531 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.536480 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" Sep 11 11:59:04.536531 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.536497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" event={"ID":"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821","Type":"ContainerDied","Data":"39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2"} Sep 11 11:59:04.536798 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.536555 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f" event={"ID":"d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821","Type":"ContainerDied","Data":"43a85eadd2b9b1078b7120f1b0975a60cc4b81a4628485b3d69fddf8f4d2464a"} Sep 11 11:59:04.536798 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.536585 2580 scope.go:117] "RemoveContainer" containerID="39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2" Sep 11 11:59:04.546762 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.546744 2580 scope.go:117] "RemoveContainer" containerID="39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2" Sep 11 11:59:04.547028 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:04.547003 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2\": container with ID starting with 39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2 not found: ID does not exist" containerID="39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2" Sep 11 11:59:04.547131 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.547057 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2"} err="failed to get container status \"39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2\": rpc error: code = NotFound desc = could not find container \"39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2\": container with ID starting with 39352e45df65e1986ec7489b22f25eefd4a1ee5bf13bd8cdf34b6f899620d9b2 not found: ID does not exist" Sep 11 11:59:04.559863 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.559843 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f"] Sep 11 11:59:04.564007 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.563986 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9def1-5774f67bc4-lls7f"] Sep 11 11:59:04.873421 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:04.873345 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" path="/var/lib/kubelet/pods/d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821/volumes" Sep 11 11:59:12.572006 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.571969 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq"] Sep 11 11:59:12.572592 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.572571 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef01991-b2f6-4f6c-b265-b55bee4f3438" containerName="test-curl-8014-cf42ef" Sep 11 11:59:12.572681 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.572594 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef01991-b2f6-4f6c-b265-b55bee4f3438" containerName="test-curl-8014-cf42ef" Sep 11 11:59:12.572681 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.572640 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" containerName="maas-api" Sep 11 11:59:12.572681 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.572650 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" containerName="maas-api" Sep 11 11:59:12.572838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.572758 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a23fa5-6d4a-4ae4-b40a-c2a8745ff821" containerName="maas-api" Sep 11 11:59:12.572838 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.572774 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef01991-b2f6-4f6c-b265-b55bee4f3438" containerName="test-curl-8014-cf42ef" Sep 11 11:59:12.577397 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.577376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.580507 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.580489 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-272c5e\"/\"rate-test-model-272c5e-kserve-self-signed-certs\"" Sep 11 11:59:12.592830 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.592802 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq"] Sep 11 11:59:12.616639 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.616606 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-tmp-dir\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.616773 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.616649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs64k\" (UniqueName: \"kubernetes.io/projected/942738b4-ce47-43e3-896c-f1e0e67c5efe-kube-api-access-bs64k\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.616773 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.616667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942738b4-ce47-43e3-896c-f1e0e67c5efe-tls-certs\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.616899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.616794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-dshm\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.616899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.616859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-home\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.616899 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.616889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-model-cache\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.718614 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.718575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-tmp-dir\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.718830 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.718646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs64k\" (UniqueName: \"kubernetes.io/projected/942738b4-ce47-43e3-896c-f1e0e67c5efe-kube-api-access-bs64k\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.718830 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.718677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942738b4-ce47-43e3-896c-f1e0e67c5efe-tls-certs\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.718830 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.718753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-dshm\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.718830 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.718814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-home\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.719115 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.718842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-model-cache\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.720414 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.719316 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-model-cache\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.720414 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.719608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-tmp-dir\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.720414 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.720182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-home\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.723074 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.723049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-dshm\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.723377 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.723359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942738b4-ce47-43e3-896c-f1e0e67c5efe-tls-certs\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.733544 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.733516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs64k\" (UniqueName: \"kubernetes.io/projected/942738b4-ce47-43e3-896c-f1e0e67c5efe-kube-api-access-bs64k\") pod \"rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:12.841393 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.841307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:59:12.888345 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:12.888318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:13.166368 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:13.166341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq"] Sep 11 11:59:13.593179 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:13.593072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" event={"ID":"942738b4-ce47-43e3-896c-f1e0e67c5efe","Type":"ContainerStarted","Data":"67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902"} Sep 11 11:59:13.593179 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:13.593124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" event={"ID":"942738b4-ce47-43e3-896c-f1e0e67c5efe","Type":"ContainerStarted","Data":"59d2dff3b169d509a78a0fe46418fa5b4626543d49d3d442d3b67531dae34786"} Sep 11 11:59:13.609735 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:13.609682 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" podStartSLOduration=1.609665527 podStartE2EDuration="1.609665527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 11:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 11:59:13.609540594 +0000 UTC m=+1515.244133727" watchObservedRunningTime="2026-09-11 11:59:13.609665527 +0000 UTC m=+1515.244258659" Sep 11 11:59:15.163736 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.163689 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" podUID="3209ad4a-2d34-4975-8b35-1dce052a2ddd" containerName="main" containerID="cri-o://79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e" gracePeriod=45 Sep 11 11:59:15.412322 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.412293 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:59:15.442166 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442089 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tmp-dir\") pod \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " Sep 11 11:59:15.442166 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442147 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tls-certs\") pod \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " Sep 11 11:59:15.442371 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442194 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzpgt\" (UniqueName: \"kubernetes.io/projected/3209ad4a-2d34-4975-8b35-1dce052a2ddd-kube-api-access-rzpgt\") pod \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " Sep 11 11:59:15.442371 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442216 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-model-cache\") pod \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " Sep 11 11:59:15.442371 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442256 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-home\") pod \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " Sep 11 11:59:15.442371 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442318 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-dshm\") pod \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\" (UID: \"3209ad4a-2d34-4975-8b35-1dce052a2ddd\") " Sep 11 11:59:15.442371 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442322 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3209ad4a-2d34-4975-8b35-1dce052a2ddd" (UID: "3209ad4a-2d34-4975-8b35-1dce052a2ddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:15.442626 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442518 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-home" (OuterVolumeSpecName: "home") pod "3209ad4a-2d34-4975-8b35-1dce052a2ddd" (UID: "3209ad4a-2d34-4975-8b35-1dce052a2ddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:15.442626 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442561 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3209ad4a-2d34-4975-8b35-1dce052a2ddd" (UID: "3209ad4a-2d34-4975-8b35-1dce052a2ddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:15.442725 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442699 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:15.442725 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442723 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:15.442898 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.442735 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:15.444488 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.444465 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3209ad4a-2d34-4975-8b35-1dce052a2ddd-kube-api-access-rzpgt" (OuterVolumeSpecName: "kube-api-access-rzpgt") pod "3209ad4a-2d34-4975-8b35-1dce052a2ddd" (UID: "3209ad4a-2d34-4975-8b35-1dce052a2ddd"). InnerVolumeSpecName "kube-api-access-rzpgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:59:15.444585 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.444503 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-dshm" (OuterVolumeSpecName: "dshm") pod "3209ad4a-2d34-4975-8b35-1dce052a2ddd" (UID: "3209ad4a-2d34-4975-8b35-1dce052a2ddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:15.444850 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.444827 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3209ad4a-2d34-4975-8b35-1dce052a2ddd" (UID: "3209ad4a-2d34-4975-8b35-1dce052a2ddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:59:15.543985 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.543959 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3209ad4a-2d34-4975-8b35-1dce052a2ddd-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:15.543985 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.543981 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpgt\" (UniqueName: \"kubernetes.io/projected/3209ad4a-2d34-4975-8b35-1dce052a2ddd-kube-api-access-rzpgt\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:15.544165 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.543991 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3209ad4a-2d34-4975-8b35-1dce052a2ddd-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:15.608935 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.608900 2580 generic.go:356] "Generic (PLEG): container finished" podID="3209ad4a-2d34-4975-8b35-1dce052a2ddd" containerID="79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e" exitCode=0 Sep 11 11:59:15.609105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.608948 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" Sep 11 11:59:15.609105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.608983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" event={"ID":"3209ad4a-2d34-4975-8b35-1dce052a2ddd","Type":"ContainerDied","Data":"79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e"} Sep 11 11:59:15.609105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.609028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb" event={"ID":"3209ad4a-2d34-4975-8b35-1dce052a2ddd","Type":"ContainerDied","Data":"72bd3a872a43ade258aa1897c40e5d6a5f1179bb28e1dbe736d622e460358d63"} Sep 11 11:59:15.609105 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.609073 2580 scope.go:117] "RemoveContainer" containerID="79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e" Sep 11 11:59:15.619407 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.619390 2580 scope.go:117] "RemoveContainer" containerID="79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e" Sep 11 11:59:15.619689 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:15.619669 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e\": container with ID starting with 79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e not found: ID does not exist" containerID="79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e" Sep 11 11:59:15.619731 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.619698 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e"} err="failed to get container status \"79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e\": rpc error: code = NotFound desc = could not find container \"79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e\": container with ID starting with 79c6906a21a1b6fc6ae63a523a34df67617cb03b5ccb0b42c78273ddccd89b3e not found: ID does not exist" Sep 11 11:59:15.630887 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.630840 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb"] Sep 11 11:59:15.634815 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:15.634790 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/sub-test-model-272c5e-kserve-5db7448df9-jnjpb"] Sep 11 11:59:16.136187 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.136130 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" podUID="06bda71c-b34d-41ee-863a-f53dc8f7e229" containerName="main" containerID="cri-o://566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a" gracePeriod=45 Sep 11 11:59:16.398747 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.398730 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:59:16.451891 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.451856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-tmp-dir\") pod \"06bda71c-b34d-41ee-863a-f53dc8f7e229\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " Sep 11 11:59:16.452081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.451922 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-home\") pod \"06bda71c-b34d-41ee-863a-f53dc8f7e229\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " Sep 11 11:59:16.452081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.451966 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bda71c-b34d-41ee-863a-f53dc8f7e229-tls-certs\") pod \"06bda71c-b34d-41ee-863a-f53dc8f7e229\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " Sep 11 11:59:16.452081 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452073 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-model-cache\") pod \"06bda71c-b34d-41ee-863a-f53dc8f7e229\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " Sep 11 11:59:16.452265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452114 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bldl2\" (UniqueName: \"kubernetes.io/projected/06bda71c-b34d-41ee-863a-f53dc8f7e229-kube-api-access-bldl2\") pod \"06bda71c-b34d-41ee-863a-f53dc8f7e229\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " Sep 11 11:59:16.452265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452173 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-dshm\") pod \"06bda71c-b34d-41ee-863a-f53dc8f7e229\" (UID: \"06bda71c-b34d-41ee-863a-f53dc8f7e229\") " Sep 11 11:59:16.452265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452175 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06bda71c-b34d-41ee-863a-f53dc8f7e229" (UID: "06bda71c-b34d-41ee-863a-f53dc8f7e229"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:16.452265 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452185 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-home" (OuterVolumeSpecName: "home") pod "06bda71c-b34d-41ee-863a-f53dc8f7e229" (UID: "06bda71c-b34d-41ee-863a-f53dc8f7e229"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:16.452432 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-model-cache" (OuterVolumeSpecName: "model-cache") pod "06bda71c-b34d-41ee-863a-f53dc8f7e229" (UID: "06bda71c-b34d-41ee-863a-f53dc8f7e229"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:16.452547 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452526 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:16.452547 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452545 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:16.452685 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.452554 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:16.454496 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.454476 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-dshm" (OuterVolumeSpecName: "dshm") pod "06bda71c-b34d-41ee-863a-f53dc8f7e229" (UID: "06bda71c-b34d-41ee-863a-f53dc8f7e229"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 11:59:16.454802 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.454777 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bda71c-b34d-41ee-863a-f53dc8f7e229-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06bda71c-b34d-41ee-863a-f53dc8f7e229" (UID: "06bda71c-b34d-41ee-863a-f53dc8f7e229"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:59:16.454802 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.454797 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bda71c-b34d-41ee-863a-f53dc8f7e229-kube-api-access-bldl2" (OuterVolumeSpecName: "kube-api-access-bldl2") pod "06bda71c-b34d-41ee-863a-f53dc8f7e229" (UID: "06bda71c-b34d-41ee-863a-f53dc8f7e229"). InnerVolumeSpecName "kube-api-access-bldl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:59:16.553791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.553761 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bldl2\" (UniqueName: \"kubernetes.io/projected/06bda71c-b34d-41ee-863a-f53dc8f7e229-kube-api-access-bldl2\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:16.553791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.553788 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06bda71c-b34d-41ee-863a-f53dc8f7e229-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:16.553791 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.553797 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06bda71c-b34d-41ee-863a-f53dc8f7e229-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:16.616224 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.616199 2580 generic.go:356] "Generic (PLEG): container finished" podID="06bda71c-b34d-41ee-863a-f53dc8f7e229" containerID="566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a" exitCode=0 Sep 11 11:59:16.616357 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.616238 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" Sep 11 11:59:16.616357 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.616274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" event={"ID":"06bda71c-b34d-41ee-863a-f53dc8f7e229","Type":"ContainerDied","Data":"566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a"} Sep 11 11:59:16.616357 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.616311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm" event={"ID":"06bda71c-b34d-41ee-863a-f53dc8f7e229","Type":"ContainerDied","Data":"3a2f6f18db82adb8e632656f4b62779cfb9b2b11f8dd5c4c68e1d95887c17e28"} Sep 11 11:59:16.616357 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.616329 2580 scope.go:117] "RemoveContainer" containerID="566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a" Sep 11 11:59:16.626441 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.626424 2580 scope.go:117] "RemoveContainer" containerID="566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a" Sep 11 11:59:16.626707 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:16.626690 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a\": container with ID starting with 566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a not found: ID does not exist" containerID="566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a" Sep 11 11:59:16.626742 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.626716 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a"} err="failed to get container status \"566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a\": rpc error: code = NotFound desc = could not find container \"566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a\": container with ID starting with 566d1a321420dcda0816dfc034399e9f19fb74a2380036807468a97058680f6a not found: ID does not exist" Sep 11 11:59:16.638805 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.638770 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm"] Sep 11 11:59:16.645844 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.645823 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-117711/sub-test-model-117711-kserve-98f4f8d7f-mzzpm"] Sep 11 11:59:16.873677 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.873597 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06bda71c-b34d-41ee-863a-f53dc8f7e229" path="/var/lib/kubelet/pods/06bda71c-b34d-41ee-863a-f53dc8f7e229/volumes" Sep 11 11:59:16.874012 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:16.873998 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3209ad4a-2d34-4975-8b35-1dce052a2ddd" path="/var/lib/kubelet/pods/3209ad4a-2d34-4975-8b35-1dce052a2ddd/volumes" Sep 11 11:59:22.889225 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:22.889140 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:22.889225 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:22.889195 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:22.902151 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:22.902124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:23.670623 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:23.670589 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 11:59:24.492313 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:24.492278 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:59:27.355745 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.355709 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9"] Sep 11 11:59:27.356206 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.355925 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" podUID="ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" containerName="maas-api" containerID="cri-o://bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb" gracePeriod=30 Sep 11 11:59:27.617618 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.617558 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:59:27.689854 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.689816 2580 generic.go:356] "Generic (PLEG): container finished" podID="ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" containerID="bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb" exitCode=0 Sep 11 11:59:27.689854 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.689849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" event={"ID":"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc","Type":"ContainerDied","Data":"bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb"} Sep 11 11:59:27.690078 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.689865 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" Sep 11 11:59:27.690078 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.689901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9" event={"ID":"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc","Type":"ContainerDied","Data":"6f7f061ac112ee9c9c5baf5cd00bee6a5dde3a56892e5ae95e856def76acffed"} Sep 11 11:59:27.690078 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.689928 2580 scope.go:117] "RemoveContainer" containerID="bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb" Sep 11 11:59:27.699186 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.699171 2580 scope.go:117] "RemoveContainer" containerID="bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb" Sep 11 11:59:27.699459 ip-10-0-135-183 kubenswrapper[2580]: E0911 11:59:27.699441 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb\": container with ID starting with bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb not found: ID does not exist" containerID="bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb" Sep 11 11:59:27.699526 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.699473 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb"} err="failed to get container status \"bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb\": rpc error: code = NotFound desc = could not find container \"bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb\": container with ID starting with bc92e9d4a29a4d5de9558b0a7cb7f9a868a5e3207825b64c2cf32c8602995dfb not found: ID does not exist" Sep 11 11:59:27.758384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.758352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-metrics-certs\") pod \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " Sep 11 11:59:27.758384 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.758390 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-openshift-service-ca\") pod \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " Sep 11 11:59:27.758630 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.758416 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7jdg\" (UniqueName: \"kubernetes.io/projected/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-kube-api-access-d7jdg\") pod \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " Sep 11 11:59:27.758630 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.758494 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-maas-api-tls\") pod \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\" (UID: \"ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc\") " Sep 11 11:59:27.758824 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.758792 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" (UID: "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 11:59:27.761080 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.761024 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" (UID: "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:59:27.761329 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.761307 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-kube-api-access-d7jdg" (OuterVolumeSpecName: "kube-api-access-d7jdg") pod "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" (UID: "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc"). InnerVolumeSpecName "kube-api-access-d7jdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 11:59:27.765529 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.765488 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" (UID: "ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 11:59:27.859570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.859532 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jdg\" (UniqueName: \"kubernetes.io/projected/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-kube-api-access-d7jdg\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:27.859570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.859562 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-maas-api-tls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:27.859570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.859571 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-metrics-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:27.859570 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:27.859579 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc-openshift-service-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 11:59:28.012434 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:28.012402 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9"] Sep 11 11:59:28.017411 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:28.017386 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-641847-f954d6464-wlgl9"] Sep 11 11:59:28.873333 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:28.873300 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" path="/var/lib/kubelet/pods/ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc/volumes" Sep 11 11:59:38.904959 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:38.904930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:59:47.495940 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:47.495905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 11:59:59.097877 ip-10-0-135-183 kubenswrapper[2580]: I0911 11:59:59.097843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:00:00.134440 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134328 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv"] Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134785 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3209ad4a-2d34-4975-8b35-1dce052a2ddd" containerName="main" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134798 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3209ad4a-2d34-4975-8b35-1dce052a2ddd" containerName="main" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134805 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" containerName="maas-api" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134811 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" containerName="maas-api" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134831 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bda71c-b34d-41ee-863a-f53dc8f7e229" containerName="main" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134837 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bda71c-b34d-41ee-863a-f53dc8f7e229" containerName="main" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134917 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffdb5162-ff3a-40e7-a47d-b4ce7280f7bc" containerName="maas-api" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134925 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bda71c-b34d-41ee-863a-f53dc8f7e229" containerName="main" Sep 11 12:00:00.136748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.134932 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3209ad4a-2d34-4975-8b35-1dce052a2ddd" containerName="main" Sep 11 12:00:00.137855 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.137840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" Sep 11 12:00:00.139930 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.139911 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f46tj\"" Sep 11 12:00:00.149428 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.149407 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" Sep 11 12:00:00.152745 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.152595 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv"] Sep 11 12:00:00.429888 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.429857 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv"] Sep 11 12:00:00.431407 ip-10-0-135-183 kubenswrapper[2580]: W0911 12:00:00.431381 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b69ae61_c5cd_42b8_bc22_f37b06f25f56.slice/crio-ee501b745ff4f63b1598425e7488f439a84d17530d60f12e66ad8e83d8dc74b4 WatchSource:0}: Error finding container ee501b745ff4f63b1598425e7488f439a84d17530d60f12e66ad8e83d8dc74b4: Status 404 returned error can't find the container with id ee501b745ff4f63b1598425e7488f439a84d17530d60f12e66ad8e83d8dc74b4 Sep 11 12:00:00.883579 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.883544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" event={"ID":"7b69ae61-c5cd-42b8-bc22-f37b06f25f56","Type":"ContainerStarted","Data":"dc452593a0a05c28d5b52b8983e257777f963bcddeb3206edf91885f7706ac9b"} Sep 11 12:00:00.883755 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.883589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" event={"ID":"7b69ae61-c5cd-42b8-bc22-f37b06f25f56","Type":"ContainerStarted","Data":"ee501b745ff4f63b1598425e7488f439a84d17530d60f12e66ad8e83d8dc74b4"} Sep 11 12:00:00.899794 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:00.899756 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" podStartSLOduration=0.899741491 podStartE2EDuration="899.741491ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:00:00.897680829 +0000 UTC m=+1562.532273973" watchObservedRunningTime="2026-09-11 12:00:00.899741491 +0000 UTC m=+1562.534334624" Sep 11 12:00:01.890141 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:01.890108 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b69ae61-c5cd-42b8-bc22-f37b06f25f56" containerID="dc452593a0a05c28d5b52b8983e257777f963bcddeb3206edf91885f7706ac9b" exitCode=0 Sep 11 12:00:01.890523 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:01.890191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" event={"ID":"7b69ae61-c5cd-42b8-bc22-f37b06f25f56","Type":"ContainerDied","Data":"dc452593a0a05c28d5b52b8983e257777f963bcddeb3206edf91885f7706ac9b"} Sep 11 12:00:03.035236 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:03.035210 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" Sep 11 12:00:03.903001 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:03.902971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" event={"ID":"7b69ae61-c5cd-42b8-bc22-f37b06f25f56","Type":"ContainerDied","Data":"ee501b745ff4f63b1598425e7488f439a84d17530d60f12e66ad8e83d8dc74b4"} Sep 11 12:00:03.903001 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:03.902994 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818800-mxzkv" Sep 11 12:00:03.903225 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:03.903016 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee501b745ff4f63b1598425e7488f439a84d17530d60f12e66ad8e83d8dc74b4" Sep 11 12:00:08.604219 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:08.604187 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:00:10.476417 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:10.476385 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk"] Sep 11 12:00:10.476926 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:10.476630 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" containerID="cri-o://f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5" gracePeriod=30 Sep 11 12:00:15.741983 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.741923 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 12:00:15.823420 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823390 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-certs\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823606 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823434 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7vgp\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-kube-api-access-m7vgp\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823606 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823467 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79ab351-95a0-43b8-9110-623194a4feaa-istiod-ca-cert\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823606 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823490 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79ab351-95a0-43b8-9110-623194a4feaa-istio-podinfo\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823606 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823543 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-envoy\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823824 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823636 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-credential-socket\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823824 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823671 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-data\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823824 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823693 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-istio-token\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823824 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823739 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-socket\") pod \"d79ab351-95a0-43b8-9110-623194a4feaa\" (UID: \"d79ab351-95a0-43b8-9110-623194a4feaa\") " Sep 11 12:00:15.823824 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823753 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:15.824189 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823899 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:15.824189 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79ab351-95a0-43b8-9110-623194a4feaa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:00:15.824189 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.823978 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-data" (OuterVolumeSpecName: "istio-data") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:15.824189 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.824145 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:15.824392 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.824264 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.824392 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.824286 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d79ab351-95a0-43b8-9110-623194a4feaa-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.824392 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.824299 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.824392 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.824315 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.824392 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.824328 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.826079 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.826056 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d79ab351-95a0-43b8-9110-623194a4feaa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:00:15.826194 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.826076 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:00:15.826260 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.826246 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-kube-api-access-m7vgp" (OuterVolumeSpecName: "kube-api-access-m7vgp") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "kube-api-access-m7vgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:00:15.826300 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.826268 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d79ab351-95a0-43b8-9110-623194a4feaa" (UID: "d79ab351-95a0-43b8-9110-623194a4feaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:15.925161 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.925127 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d79ab351-95a0-43b8-9110-623194a4feaa-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.925161 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.925153 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.925161 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.925163 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vgp\" (UniqueName: \"kubernetes.io/projected/d79ab351-95a0-43b8-9110-623194a4feaa-kube-api-access-m7vgp\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.925161 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.925172 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d79ab351-95a0-43b8-9110-623194a4feaa-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:15.971664 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.971636 2580 generic.go:356] "Generic (PLEG): container finished" podID="d79ab351-95a0-43b8-9110-623194a4feaa" containerID="f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5" exitCode=0 Sep 11 12:00:15.971830 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.971680 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" Sep 11 12:00:15.971830 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.971681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" event={"ID":"d79ab351-95a0-43b8-9110-623194a4feaa","Type":"ContainerDied","Data":"f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5"} Sep 11 12:00:15.971830 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.971813 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk" event={"ID":"d79ab351-95a0-43b8-9110-623194a4feaa","Type":"ContainerDied","Data":"b3eb0d4a2641d711425c0669f31bf5f143b1208fc91fa8b6cd65fe0b6f6e388c"} Sep 11 12:00:15.971952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.971841 2580 scope.go:117] "RemoveContainer" containerID="f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5" Sep 11 12:00:15.981712 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.981698 2580 scope.go:117] "RemoveContainer" containerID="98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743" Sep 11 12:00:15.992485 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.992463 2580 scope.go:117] "RemoveContainer" containerID="f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5" Sep 11 12:00:15.992742 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:00:15.992726 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5\": container with ID starting with f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5 not found: ID does not exist" containerID="f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5" Sep 11 12:00:15.992796 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.992751 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5"} err="failed to get container status \"f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5\": rpc error: code = NotFound desc = could not find container \"f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5\": container with ID starting with f41905c63c1889f827c491a3183b7a713ff39d094a254d42adb65aab22ddbec5 not found: ID does not exist" Sep 11 12:00:15.992796 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.992766 2580 scope.go:117] "RemoveContainer" containerID="98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743" Sep 11 12:00:15.992995 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:00:15.992978 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743\": container with ID starting with 98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743 not found: ID does not exist" containerID="98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743" Sep 11 12:00:15.993084 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:15.993006 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743"} err="failed to get container status \"98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743\": rpc error: code = NotFound desc = could not find container \"98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743\": container with ID starting with 98746b15d5ad6efbac5d53ce9a371804ea652c46e542e4e112719545a8a58743 not found: ID does not exist" Sep 11 12:00:16.005235 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:16.005202 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk"] Sep 11 12:00:16.016216 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:16.011028 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6daa53-openshift-default-d4f886bd4-pjqpk"] Sep 11 12:00:16.874372 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:16.874341 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" path="/var/lib/kubelet/pods/d79ab351-95a0-43b8-9110-623194a4feaa/volumes" Sep 11 12:00:21.499146 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:21.499108 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:00:25.505660 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:25.505625 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq"] Sep 11 12:00:33.101114 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:33.101082 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:00:40.525114 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.525064 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" podUID="942738b4-ce47-43e3-896c-f1e0e67c5efe" containerName="main" containerID="cri-o://67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902" gracePeriod=45 Sep 11 12:00:40.782549 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.782492 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 12:00:40.858985 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.858962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-home\") pod \"942738b4-ce47-43e3-896c-f1e0e67c5efe\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " Sep 11 12:00:40.859160 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.858996 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-tmp-dir\") pod \"942738b4-ce47-43e3-896c-f1e0e67c5efe\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " Sep 11 12:00:40.859160 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs64k\" (UniqueName: \"kubernetes.io/projected/942738b4-ce47-43e3-896c-f1e0e67c5efe-kube-api-access-bs64k\") pod \"942738b4-ce47-43e3-896c-f1e0e67c5efe\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " Sep 11 12:00:40.859160 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859119 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-dshm\") pod \"942738b4-ce47-43e3-896c-f1e0e67c5efe\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " Sep 11 12:00:40.859279 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859189 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-model-cache\") pod \"942738b4-ce47-43e3-896c-f1e0e67c5efe\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " Sep 11 12:00:40.859279 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859220 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942738b4-ce47-43e3-896c-f1e0e67c5efe-tls-certs\") pod \"942738b4-ce47-43e3-896c-f1e0e67c5efe\" (UID: \"942738b4-ce47-43e3-896c-f1e0e67c5efe\") " Sep 11 12:00:40.859382 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859269 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "942738b4-ce47-43e3-896c-f1e0e67c5efe" (UID: "942738b4-ce47-43e3-896c-f1e0e67c5efe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:40.859382 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-home" (OuterVolumeSpecName: "home") pod "942738b4-ce47-43e3-896c-f1e0e67c5efe" (UID: "942738b4-ce47-43e3-896c-f1e0e67c5efe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:40.859489 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859428 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-model-cache" (OuterVolumeSpecName: "model-cache") pod "942738b4-ce47-43e3-896c-f1e0e67c5efe" (UID: "942738b4-ce47-43e3-896c-f1e0e67c5efe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:40.859553 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859540 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:40.859553 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859553 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:40.859645 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.859561 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:40.861384 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.861340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-dshm" (OuterVolumeSpecName: "dshm") pod "942738b4-ce47-43e3-896c-f1e0e67c5efe" (UID: "942738b4-ce47-43e3-896c-f1e0e67c5efe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:00:40.861504 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.861398 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942738b4-ce47-43e3-896c-f1e0e67c5efe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "942738b4-ce47-43e3-896c-f1e0e67c5efe" (UID: "942738b4-ce47-43e3-896c-f1e0e67c5efe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:00:40.861504 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.861460 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942738b4-ce47-43e3-896c-f1e0e67c5efe-kube-api-access-bs64k" (OuterVolumeSpecName: "kube-api-access-bs64k") pod "942738b4-ce47-43e3-896c-f1e0e67c5efe" (UID: "942738b4-ce47-43e3-896c-f1e0e67c5efe"). InnerVolumeSpecName "kube-api-access-bs64k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:00:40.960607 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.960571 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942738b4-ce47-43e3-896c-f1e0e67c5efe-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:40.960607 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.960603 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs64k\" (UniqueName: \"kubernetes.io/projected/942738b4-ce47-43e3-896c-f1e0e67c5efe-kube-api-access-bs64k\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:40.960855 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:40.960619 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942738b4-ce47-43e3-896c-f1e0e67c5efe-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:00:41.114852 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.114762 2580 generic.go:356] "Generic (PLEG): container finished" podID="942738b4-ce47-43e3-896c-f1e0e67c5efe" containerID="67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902" exitCode=0 Sep 11 12:00:41.114852 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.114821 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" Sep 11 12:00:41.115025 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.114851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" event={"ID":"942738b4-ce47-43e3-896c-f1e0e67c5efe","Type":"ContainerDied","Data":"67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902"} Sep 11 12:00:41.115025 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.114897 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq" event={"ID":"942738b4-ce47-43e3-896c-f1e0e67c5efe","Type":"ContainerDied","Data":"59d2dff3b169d509a78a0fe46418fa5b4626543d49d3d442d3b67531dae34786"} Sep 11 12:00:41.115025 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.114917 2580 scope.go:117] "RemoveContainer" containerID="67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902" Sep 11 12:00:41.124332 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.124318 2580 scope.go:117] "RemoveContainer" containerID="67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902" Sep 11 12:00:41.124624 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:00:41.124600 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902\": container with ID starting with 67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902 not found: ID does not exist" containerID="67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902" Sep 11 12:00:41.124729 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.124630 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902"} err="failed to get container status \"67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902\": rpc error: code = NotFound desc = could not find container \"67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902\": container with ID starting with 67f1ee3eb3a74cf00ca18a4c7c28b402b4f5cb15b3e16305968037107e31b902 not found: ID does not exist" Sep 11 12:00:41.133232 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.133210 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq"] Sep 11 12:00:41.138399 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:41.138380 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-272c5e/rate-test-model-272c5e-kserve-5d796bbcbc-cj6gq"] Sep 11 12:00:42.872708 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:42.872676 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942738b4-ce47-43e3-896c-f1e0e67c5efe" path="/var/lib/kubelet/pods/942738b4-ce47-43e3-896c-f1e0e67c5efe/volumes" Sep 11 12:00:43.800548 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:43.800517 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:00:56.352919 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.352839 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt"] Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353314 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942738b4-ce47-43e3-896c-f1e0e67c5efe" containerName="main" Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353327 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="942738b4-ce47-43e3-896c-f1e0e67c5efe" containerName="main" Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353335 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353341 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353354 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b69ae61-c5cd-42b8-bc22-f37b06f25f56" containerName="cleanup" Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353361 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b69ae61-c5cd-42b8-bc22-f37b06f25f56" containerName="cleanup" Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353381 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" Sep 11 12:00:56.353387 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353387 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" Sep 11 12:00:56.353947 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353465 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" Sep 11 12:00:56.353947 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353476 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79ab351-95a0-43b8-9110-623194a4feaa" containerName="istio-proxy" Sep 11 12:00:56.353947 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353483 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b69ae61-c5cd-42b8-bc22-f37b06f25f56" containerName="cleanup" Sep 11 12:00:56.353947 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.353491 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="942738b4-ce47-43e3-896c-f1e0e67c5efe" containerName="main" Sep 11 12:00:56.358015 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.357999 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.366124 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.366093 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-720006-openshift-default-dockercfg-f7t4r\"" Sep 11 12:00:56.391342 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.391309 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt"] Sep 11 12:00:56.510670 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-credential-socket\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.510670 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-envoy\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.510893 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-data\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.510893 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510715 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-token\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.510893 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e31339-c020-4ac0-a3b8-742ea9d01423-istiod-ca-cert\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.510893 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-certs\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.511027 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-socket\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.511027 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwtrh\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-kube-api-access-lwtrh\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.511027 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.510986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-podinfo\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.611950 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.611850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-credential-socket\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.611950 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.611897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-envoy\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.611950 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.611915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-data\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.611950 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.611931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-token\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.611950 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.611955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e31339-c020-4ac0-a3b8-742ea9d01423-istiod-ca-cert\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612404 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-certs\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612404 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-socket\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612404 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwtrh\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-kube-api-access-lwtrh\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612404 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-podinfo\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612404 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-data\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612404 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612382 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-credential-socket\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612757 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612571 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-socket\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612757 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612622 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-certs\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.612886 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.612862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e31339-c020-4ac0-a3b8-742ea9d01423-istiod-ca-cert\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.614399 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.614382 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-envoy\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.614818 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.614803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-podinfo\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.623468 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.623436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwtrh\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-kube-api-access-lwtrh\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.624848 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.624829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-token\") pod \"e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.669088 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.669024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:56.965821 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.965791 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt"] Sep 11 12:00:56.967898 ip-10-0-135-183 kubenswrapper[2580]: W0911 12:00:56.967872 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e31339_c020_4ac0_a3b8_742ea9d01423.slice/crio-b88e3dffd7adc45e4d702444c163a3821cc3ecd11ea6712835e6d479b141b990 WatchSource:0}: Error finding container b88e3dffd7adc45e4d702444c163a3821cc3ecd11ea6712835e6d479b141b990: Status 404 returned error can't find the container with id b88e3dffd7adc45e4d702444c163a3821cc3ecd11ea6712835e6d479b141b990 Sep 11 12:00:56.970236 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.970203 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 12:00:56.970346 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.970280 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 12:00:56.970346 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:56.970324 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 12:00:57.209172 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:57.209138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" event={"ID":"13e31339-c020-4ac0-a3b8-742ea9d01423","Type":"ContainerStarted","Data":"0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b"} Sep 11 12:00:57.209376 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:57.209186 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" event={"ID":"13e31339-c020-4ac0-a3b8-742ea9d01423","Type":"ContainerStarted","Data":"b88e3dffd7adc45e4d702444c163a3821cc3ecd11ea6712835e6d479b141b990"} Sep 11 12:00:57.253375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:57.253263 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podStartSLOduration=1.253241954 podStartE2EDuration="1.253241954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:00:57.251704212 +0000 UTC m=+1618.886297371" watchObservedRunningTime="2026-09-11 12:00:57.253241954 +0000 UTC m=+1618.887835088" Sep 11 12:00:57.669425 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:57.669396 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:58.674363 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:58.674336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:59.220486 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:59.220437 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:00:59.221449 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:00:59.221434 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:01:00.341160 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.341125 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9"] Sep 11 12:01:00.346111 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.346091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.349250 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.349229 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-720006-serving-cert\"" Sep 11 12:01:00.365426 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.365404 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9"] Sep 11 12:01:00.447237 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.447211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.447380 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.447247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.447380 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.447304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-metrics-certs\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.447380 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.447342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dn8s\" (UniqueName: \"kubernetes.io/projected/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-kube-api-access-6dn8s\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.547910 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.547861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-metrics-certs\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.548155 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.547932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dn8s\" (UniqueName: \"kubernetes.io/projected/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-kube-api-access-6dn8s\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.548155 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.547976 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.548155 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.548000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.548155 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:01:00.548110 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-serving-cert: secret "maas-api-e2e-ipp-b-720006-serving-cert" not found Sep 11 12:01:00.548402 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:01:00.548175 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls podName:9017fc15-d16a-4de7-9c0d-60ab92a66c4c nodeName:}" failed. No retries permitted until 2026-09-11 12:01:01.04815681 +0000 UTC m=+1622.682749920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls") pod "maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" (UID: "9017fc15-d16a-4de7-9c0d-60ab92a66c4c") : secret "maas-api-e2e-ipp-b-720006-serving-cert" not found Sep 11 12:01:00.548625 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.548608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.550631 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.550608 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-metrics-certs\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:00.556663 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:00.556645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dn8s\" (UniqueName: \"kubernetes.io/projected/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-kube-api-access-6dn8s\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:01.052985 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:01.052951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:01.055428 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:01.055404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-720006-f944ff868-d6gt9\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:01.257372 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:01.257339 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:01.534071 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:01.534027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9"] Sep 11 12:01:01.536439 ip-10-0-135-183 kubenswrapper[2580]: W0911 12:01:01.536407 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9017fc15_d16a_4de7_9c0d_60ab92a66c4c.slice/crio-7203cf4771b74596eebec921b68ad6789d00786f7014d6dc70b9111354c090f7 WatchSource:0}: Error finding container 7203cf4771b74596eebec921b68ad6789d00786f7014d6dc70b9111354c090f7: Status 404 returned error can't find the container with id 7203cf4771b74596eebec921b68ad6789d00786f7014d6dc70b9111354c090f7 Sep 11 12:01:01.537966 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:01.537948 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 12:01:02.239931 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:02.239899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" event={"ID":"9017fc15-d16a-4de7-9c0d-60ab92a66c4c","Type":"ContainerStarted","Data":"7203cf4771b74596eebec921b68ad6789d00786f7014d6dc70b9111354c090f7"} Sep 11 12:01:03.247507 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:03.247479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" event={"ID":"9017fc15-d16a-4de7-9c0d-60ab92a66c4c","Type":"ContainerStarted","Data":"2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6"} Sep 11 12:01:03.247952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:03.247601 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:03.269747 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:03.269689 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" podStartSLOduration=2.065390732 podStartE2EDuration="3.269667819s" totalImagesPullingTime="1.204277087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:01:03.265111569 +0000 UTC m=+1624.899704702" watchObservedRunningTime="2026-09-11 12:01:03.269667819 +0000 UTC m=+1624.904260953" Sep 11 12:01:05.808140 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:05.808105 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:01:09.273335 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:09.273302 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:01:15.262211 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:15.262172 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 12:01:15.262596 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:15.262239 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 12:01:16.336159 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:16.336133 2580 generic.go:356] "Generic (PLEG): container finished" podID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerID="0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b" exitCode=0 Sep 11 12:01:16.336536 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:16.336220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" event={"ID":"13e31339-c020-4ac0-a3b8-742ea9d01423","Type":"ContainerDied","Data":"0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b"} Sep 11 12:01:16.336666 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:16.336651 2580 scope.go:117] "RemoveContainer" containerID="0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b" Sep 11 12:01:16.337703 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:16.337674 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 12:01:16.337801 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:16.337758 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 12:01:16.337863 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:16.337809 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 12:01:17.343789 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:17.343753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" event={"ID":"13e31339-c020-4ac0-a3b8-742ea9d01423","Type":"ContainerStarted","Data":"f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159"} Sep 11 12:01:17.669688 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:17.669656 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:01:17.670094 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:17.669984 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 12:01:17.670337 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:17.670314 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 12:01:18.669943 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:18.669888 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 12:01:18.670462 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:18.669974 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 12:01:19.670277 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:19.670238 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 12:01:19.670862 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:19.670828 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 12:01:20.673528 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:20.673485 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 12:01:20.674141 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:20.674106 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 12:01:21.670306 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:21.670263 2580 patch_prober.go:28] interesting pod/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 12:01:21.670524 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:21.670336 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 12:01:22.673164 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:22.673134 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:01:22.673531 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:22.673374 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:01:22.674211 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:22.674193 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:01:36.105587 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.105543 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v"] Sep 11 12:01:36.109315 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.109292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.111503 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.111482 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b19164\"/\"openshift-service-ca.crt\"" Sep 11 12:01:36.112225 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.112203 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b19164\"/\"default-dockercfg-68pzz\"" Sep 11 12:01:36.112350 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.112261 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b19164\"/\"kube-root-ca.crt\"" Sep 11 12:01:36.112350 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.112292 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b19164\"/\"ipp-route-b19164-kserve-self-signed-certs\"" Sep 11 12:01:36.117483 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.117462 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v"] Sep 11 12:01:36.191220 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.191187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tmp-dir\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.191392 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.191243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-home\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.191392 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.191359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tls-certs\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.191490 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.191402 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-model-cache\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.191490 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.191444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p64l\" (UniqueName: \"kubernetes.io/projected/9a982545-f86c-4ab1-8ad4-3c038321e8a9-kube-api-access-7p64l\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.191579 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.191554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-dshm\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.292609 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.292578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-dshm\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.292804 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.292631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tmp-dir\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.292804 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.292666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-home\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.292804 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.292714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tls-certs\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.292944 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.292838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-model-cache\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.292944 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.292909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p64l\" (UniqueName: \"kubernetes.io/projected/9a982545-f86c-4ab1-8ad4-3c038321e8a9-kube-api-access-7p64l\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.293180 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.293144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tmp-dir\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.293323 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.293214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-model-cache\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.293560 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.293361 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-home\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.295195 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.295163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-dshm\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.295301 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.295275 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tls-certs\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.301191 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.301172 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p64l\" (UniqueName: \"kubernetes.io/projected/9a982545-f86c-4ab1-8ad4-3c038321e8a9-kube-api-access-7p64l\") pod \"ipp-route-b19164-kserve-5c8fc87799-v7p5v\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.420142 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.420112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:36.699961 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:36.699933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v"] Sep 11 12:01:36.701597 ip-10-0-135-183 kubenswrapper[2580]: W0911 12:01:36.701571 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a982545_f86c_4ab1_8ad4_3c038321e8a9.slice/crio-7dd48ecfaf75f240263965760bc747bfb4331d8912a9e6ac3b6193060a8af7df WatchSource:0}: Error finding container 7dd48ecfaf75f240263965760bc747bfb4331d8912a9e6ac3b6193060a8af7df: Status 404 returned error can't find the container with id 7dd48ecfaf75f240263965760bc747bfb4331d8912a9e6ac3b6193060a8af7df Sep 11 12:01:37.462198 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:37.462161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" event={"ID":"9a982545-f86c-4ab1-8ad4-3c038321e8a9","Type":"ContainerStarted","Data":"78890a55d2b737bff48a18a2a4da2d766f8b4aa89fc1312b9dc056de11aabb3e"} Sep 11 12:01:37.462610 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:37.462215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" event={"ID":"9a982545-f86c-4ab1-8ad4-3c038321e8a9","Type":"ContainerStarted","Data":"7dd48ecfaf75f240263965760bc747bfb4331d8912a9e6ac3b6193060a8af7df"} Sep 11 12:01:37.480375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:37.480331 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" podStartSLOduration=1.480317069 podStartE2EDuration="1.480317069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:01:37.478021474 +0000 UTC m=+1659.112614609" watchObservedRunningTime="2026-09-11 12:01:37.480317069 +0000 UTC m=+1659.114910201" Sep 11 12:01:39.900293 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:39.900257 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:01:46.420489 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:46.420457 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:46.420914 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:46.420694 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:46.433212 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:46.433190 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:46.524332 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:46.524305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:01:53.396447 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:01:53.396413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:02:11.734123 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:02:11.734087 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:02:39.906169 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:02:39.906136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:02:59.610722 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:02:59.610685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:03:07.820128 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:07.820092 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v"] Sep 11 12:03:13.017961 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:13.017926 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:03:22.842056 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:22.841974 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" podUID="9a982545-f86c-4ab1-8ad4-3c038321e8a9" containerName="main" containerID="cri-o://78890a55d2b737bff48a18a2a4da2d766f8b4aa89fc1312b9dc056de11aabb3e" gracePeriod=45 Sep 11 12:03:23.068579 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.068547 2580 generic.go:356] "Generic (PLEG): container finished" podID="9a982545-f86c-4ab1-8ad4-3c038321e8a9" containerID="78890a55d2b737bff48a18a2a4da2d766f8b4aa89fc1312b9dc056de11aabb3e" exitCode=0 Sep 11 12:03:23.068736 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.068620 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" event={"ID":"9a982545-f86c-4ab1-8ad4-3c038321e8a9","Type":"ContainerDied","Data":"78890a55d2b737bff48a18a2a4da2d766f8b4aa89fc1312b9dc056de11aabb3e"} Sep 11 12:03:23.105306 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.105254 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:03:23.251278 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251248 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p64l\" (UniqueName: \"kubernetes.io/projected/9a982545-f86c-4ab1-8ad4-3c038321e8a9-kube-api-access-7p64l\") pod \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " Sep 11 12:03:23.251445 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251300 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tmp-dir\") pod \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " Sep 11 12:03:23.251445 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251343 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-dshm\") pod \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " Sep 11 12:03:23.251445 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251407 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-model-cache\") pod \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " Sep 11 12:03:23.251604 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251491 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tls-certs\") pod \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " Sep 11 12:03:23.251604 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251538 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-home\") pod \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\" (UID: \"9a982545-f86c-4ab1-8ad4-3c038321e8a9\") " Sep 11 12:03:23.251712 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251642 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a982545-f86c-4ab1-8ad4-3c038321e8a9" (UID: "9a982545-f86c-4ab1-8ad4-3c038321e8a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:03:23.251771 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251728 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a982545-f86c-4ab1-8ad4-3c038321e8a9" (UID: "9a982545-f86c-4ab1-8ad4-3c038321e8a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:03:23.251873 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251838 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-home" (OuterVolumeSpecName: "home") pod "9a982545-f86c-4ab1-8ad4-3c038321e8a9" (UID: "9a982545-f86c-4ab1-8ad4-3c038321e8a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:03:23.252010 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251964 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-home\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:03:23.252010 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.251986 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tmp-dir\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:03:23.252322 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.252212 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-model-cache\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:03:23.253705 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.253684 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-dshm" (OuterVolumeSpecName: "dshm") pod "9a982545-f86c-4ab1-8ad4-3c038321e8a9" (UID: "9a982545-f86c-4ab1-8ad4-3c038321e8a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:03:23.253779 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.253712 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a982545-f86c-4ab1-8ad4-3c038321e8a9" (UID: "9a982545-f86c-4ab1-8ad4-3c038321e8a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:03:23.253867 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.253846 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a982545-f86c-4ab1-8ad4-3c038321e8a9-kube-api-access-7p64l" (OuterVolumeSpecName: "kube-api-access-7p64l") pod "9a982545-f86c-4ab1-8ad4-3c038321e8a9" (UID: "9a982545-f86c-4ab1-8ad4-3c038321e8a9"). InnerVolumeSpecName "kube-api-access-7p64l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:03:23.353489 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.353453 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a982545-f86c-4ab1-8ad4-3c038321e8a9-tls-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:03:23.353489 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.353482 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p64l\" (UniqueName: \"kubernetes.io/projected/9a982545-f86c-4ab1-8ad4-3c038321e8a9-kube-api-access-7p64l\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:03:23.353489 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:23.353495 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a982545-f86c-4ab1-8ad4-3c038321e8a9-dshm\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:03:24.076803 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:24.076772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" event={"ID":"9a982545-f86c-4ab1-8ad4-3c038321e8a9","Type":"ContainerDied","Data":"7dd48ecfaf75f240263965760bc747bfb4331d8912a9e6ac3b6193060a8af7df"} Sep 11 12:03:24.076803 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:24.076798 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v" Sep 11 12:03:24.077305 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:24.076825 2580 scope.go:117] "RemoveContainer" containerID="78890a55d2b737bff48a18a2a4da2d766f8b4aa89fc1312b9dc056de11aabb3e" Sep 11 12:03:24.102813 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:24.102788 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v"] Sep 11 12:03:24.105128 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:24.105109 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b19164/ipp-route-b19164-kserve-5c8fc87799-v7p5v"] Sep 11 12:03:24.873748 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:24.873718 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a982545-f86c-4ab1-8ad4-3c038321e8a9" path="/var/lib/kubelet/pods/9a982545-f86c-4ab1-8ad4-3c038321e8a9/volumes" Sep 11 12:03:59.026979 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:59.026949 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 12:03:59.026979 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:03:59.026961 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 12:04:03.677934 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:03.677901 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9"] Sep 11 12:04:03.678535 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:03.678133 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" podUID="9017fc15-d16a-4de7-9c0d-60ab92a66c4c" containerName="maas-api" containerID="cri-o://2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6" gracePeriod=30 Sep 11 12:04:03.944970 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:03.944899 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:04:04.028331 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.028303 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-openshift-service-ca\") pod \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " Sep 11 12:04:04.028331 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.028334 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-metrics-certs\") pod \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " Sep 11 12:04:04.028600 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.028358 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dn8s\" (UniqueName: \"kubernetes.io/projected/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-kube-api-access-6dn8s\") pod \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " Sep 11 12:04:04.028600 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.028566 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls\") pod \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\" (UID: \"9017fc15-d16a-4de7-9c0d-60ab92a66c4c\") " Sep 11 12:04:04.028772 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.028748 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9017fc15-d16a-4de7-9c0d-60ab92a66c4c" (UID: "9017fc15-d16a-4de7-9c0d-60ab92a66c4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:04:04.028983 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.028963 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-openshift-service-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:04.030723 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.030698 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9017fc15-d16a-4de7-9c0d-60ab92a66c4c" (UID: "9017fc15-d16a-4de7-9c0d-60ab92a66c4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:04:04.030723 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.030717 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9017fc15-d16a-4de7-9c0d-60ab92a66c4c" (UID: "9017fc15-d16a-4de7-9c0d-60ab92a66c4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:04:04.030850 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.030794 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-kube-api-access-6dn8s" (OuterVolumeSpecName: "kube-api-access-6dn8s") pod "9017fc15-d16a-4de7-9c0d-60ab92a66c4c" (UID: "9017fc15-d16a-4de7-9c0d-60ab92a66c4c"). InnerVolumeSpecName "kube-api-access-6dn8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:04:04.130486 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.130442 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-maas-api-tls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:04.130486 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.130488 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-metrics-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:04.130698 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.130503 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dn8s\" (UniqueName: \"kubernetes.io/projected/9017fc15-d16a-4de7-9c0d-60ab92a66c4c-kube-api-access-6dn8s\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:04.314855 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.314761 2580 generic.go:356] "Generic (PLEG): container finished" podID="9017fc15-d16a-4de7-9c0d-60ab92a66c4c" containerID="2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6" exitCode=0 Sep 11 12:04:04.314855 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.314821 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" Sep 11 12:04:04.315105 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.314848 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" event={"ID":"9017fc15-d16a-4de7-9c0d-60ab92a66c4c","Type":"ContainerDied","Data":"2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6"} Sep 11 12:04:04.315105 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.314901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9" event={"ID":"9017fc15-d16a-4de7-9c0d-60ab92a66c4c","Type":"ContainerDied","Data":"7203cf4771b74596eebec921b68ad6789d00786f7014d6dc70b9111354c090f7"} Sep 11 12:04:04.315105 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.314925 2580 scope.go:117] "RemoveContainer" containerID="2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6" Sep 11 12:04:04.324983 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.324967 2580 scope.go:117] "RemoveContainer" containerID="2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6" Sep 11 12:04:04.325287 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:04:04.325265 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6\": container with ID starting with 2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6 not found: ID does not exist" containerID="2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6" Sep 11 12:04:04.325352 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.325297 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6"} err="failed to get container status \"2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6\": rpc error: code = NotFound desc = could not find container \"2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6\": container with ID starting with 2653e92c0173b43e514e2ffc76b6cab819381c80b049292b4f2505bd4750a7c6 not found: ID does not exist" Sep 11 12:04:04.341161 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.341131 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9"] Sep 11 12:04:04.350455 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.350361 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-720006-f944ff868-d6gt9"] Sep 11 12:04:04.873021 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:04.872985 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9017fc15-d16a-4de7-9c0d-60ab92a66c4c" path="/var/lib/kubelet/pods/9017fc15-d16a-4de7-9c0d-60ab92a66c4c/volumes" Sep 11 12:04:06.733259 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:06.733225 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:04:11.532442 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:11.532400 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt"] Sep 11 12:04:11.532937 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:11.532675 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" containerID="cri-o://f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159" gracePeriod=30 Sep 11 12:04:16.798021 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.797961 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:04:16.951475 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951440 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-data\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951475 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951484 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-socket\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951685 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951508 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-credential-socket\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951685 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951531 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-podinfo\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951685 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951597 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwtrh\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-kube-api-access-lwtrh\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951685 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951646 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e31339-c020-4ac0-a3b8-742ea9d01423-istiod-ca-cert\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951884 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951695 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-envoy\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951884 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951734 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-token\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951884 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951795 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-certs\") pod \"13e31339-c020-4ac0-a3b8-742ea9d01423\" (UID: \"13e31339-c020-4ac0-a3b8-742ea9d01423\") " Sep 11 12:04:16.951884 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951767 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-data" (OuterVolumeSpecName: "istio-data") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:16.951884 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951833 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:16.952177 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.951912 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:16.952177 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.952114 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:16.952177 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.952127 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:16.952177 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.952135 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:16.952177 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.952158 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:16.952452 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.952187 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e31339-c020-4ac0-a3b8-742ea9d01423-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:04:16.954110 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.954082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:16.954110 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.954088 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:04:16.954275 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.954123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-kube-api-access-lwtrh" (OuterVolumeSpecName: "kube-api-access-lwtrh") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "kube-api-access-lwtrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:04:16.954422 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:16.954406 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-token" (OuterVolumeSpecName: "istio-token") pod "13e31339-c020-4ac0-a3b8-742ea9d01423" (UID: "13e31339-c020-4ac0-a3b8-742ea9d01423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:04:17.053727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.053615 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:17.053727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.053653 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:17.053727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.053667 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwtrh\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-kube-api-access-lwtrh\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:17.053727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.053681 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e31339-c020-4ac0-a3b8-742ea9d01423-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:17.053727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.053694 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:17.053727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.053706 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e31339-c020-4ac0-a3b8-742ea9d01423-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:17.392570 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.392470 2580 generic.go:356] "Generic (PLEG): container finished" podID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerID="f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159" exitCode=0 Sep 11 12:04:17.392570 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.392518 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" Sep 11 12:04:17.392570 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.392557 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" event={"ID":"13e31339-c020-4ac0-a3b8-742ea9d01423","Type":"ContainerDied","Data":"f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159"} Sep 11 12:04:17.392825 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.392611 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt" event={"ID":"13e31339-c020-4ac0-a3b8-742ea9d01423","Type":"ContainerDied","Data":"b88e3dffd7adc45e4d702444c163a3821cc3ecd11ea6712835e6d479b141b990"} Sep 11 12:04:17.392825 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.392633 2580 scope.go:117] "RemoveContainer" containerID="f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159" Sep 11 12:04:17.402703 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.402685 2580 scope.go:117] "RemoveContainer" containerID="0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b" Sep 11 12:04:17.411344 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.411327 2580 scope.go:117] "RemoveContainer" containerID="f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159" Sep 11 12:04:17.411602 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:04:17.411584 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159\": container with ID starting with f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159 not found: ID does not exist" containerID="f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159" Sep 11 12:04:17.411646 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.411612 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159"} err="failed to get container status \"f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159\": rpc error: code = NotFound desc = could not find container \"f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159\": container with ID starting with f176a66ba576145500995f5b73e841f864b76cf945b35d9078ceecc7f235e159 not found: ID does not exist" Sep 11 12:04:17.411646 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.411629 2580 scope.go:117] "RemoveContainer" containerID="0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b" Sep 11 12:04:17.411871 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:04:17.411854 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b\": container with ID starting with 0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b not found: ID does not exist" containerID="0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b" Sep 11 12:04:17.411926 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.411876 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b"} err="failed to get container status \"0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b\": rpc error: code = NotFound desc = could not find container \"0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b\": container with ID starting with 0e0277efd6cd1415885647509341035870d8f1fe525e1fedae1a9ca152ccb14b not found: ID does not exist" Sep 11 12:04:17.421012 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.420989 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt"] Sep 11 12:04:17.430704 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:17.430678 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-720006-openshift-default-5754749bc8-twkxt"] Sep 11 12:04:18.878839 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:18.878803 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" path="/var/lib/kubelet/pods/13e31339-c020-4ac0-a3b8-742ea9d01423/volumes" Sep 11 12:04:25.611969 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:25.611936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:04:41.964628 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:41.964589 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6"] Sep 11 12:04:41.965184 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:41.964843 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" podUID="2a8c8284-f727-43ae-854e-c60c03b0438b" containerName="maas-api" containerID="cri-o://f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af" gracePeriod=30 Sep 11 12:04:42.227198 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.227136 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 12:04:42.290083 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.290028 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-metrics-certs\") pod \"2a8c8284-f727-43ae-854e-c60c03b0438b\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " Sep 11 12:04:42.290261 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.290126 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls\") pod \"2a8c8284-f727-43ae-854e-c60c03b0438b\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " Sep 11 12:04:42.290261 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.290177 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c8284-f727-43ae-854e-c60c03b0438b-openshift-service-ca\") pod \"2a8c8284-f727-43ae-854e-c60c03b0438b\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " Sep 11 12:04:42.290261 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.290232 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5dmg\" (UniqueName: \"kubernetes.io/projected/2a8c8284-f727-43ae-854e-c60c03b0438b-kube-api-access-v5dmg\") pod \"2a8c8284-f727-43ae-854e-c60c03b0438b\" (UID: \"2a8c8284-f727-43ae-854e-c60c03b0438b\") " Sep 11 12:04:42.290612 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.290588 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8c8284-f727-43ae-854e-c60c03b0438b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a8c8284-f727-43ae-854e-c60c03b0438b" (UID: "2a8c8284-f727-43ae-854e-c60c03b0438b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:04:42.292447 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.292420 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8c8284-f727-43ae-854e-c60c03b0438b-kube-api-access-v5dmg" (OuterVolumeSpecName: "kube-api-access-v5dmg") pod "2a8c8284-f727-43ae-854e-c60c03b0438b" (UID: "2a8c8284-f727-43ae-854e-c60c03b0438b"). InnerVolumeSpecName "kube-api-access-v5dmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:04:42.292882 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.292864 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a8c8284-f727-43ae-854e-c60c03b0438b" (UID: "2a8c8284-f727-43ae-854e-c60c03b0438b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:04:42.292969 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.292880 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2a8c8284-f727-43ae-854e-c60c03b0438b" (UID: "2a8c8284-f727-43ae-854e-c60c03b0438b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:04:42.391737 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.391700 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5dmg\" (UniqueName: \"kubernetes.io/projected/2a8c8284-f727-43ae-854e-c60c03b0438b-kube-api-access-v5dmg\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:42.391737 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.391732 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-metrics-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:42.391737 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.391742 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a8c8284-f727-43ae-854e-c60c03b0438b-maas-api-tls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:42.391965 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.391750 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a8c8284-f727-43ae-854e-c60c03b0438b-openshift-service-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:42.542979 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.542893 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a8c8284-f727-43ae-854e-c60c03b0438b" containerID="f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af" exitCode=0 Sep 11 12:04:42.542979 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.542946 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" Sep 11 12:04:42.543211 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.542949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" event={"ID":"2a8c8284-f727-43ae-854e-c60c03b0438b","Type":"ContainerDied","Data":"f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af"} Sep 11 12:04:42.543211 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.543089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6" event={"ID":"2a8c8284-f727-43ae-854e-c60c03b0438b","Type":"ContainerDied","Data":"5bbc843bb83ed79177e7477e80207d315270f8a022b0ee75b92e994ef74b3bb3"} Sep 11 12:04:42.543211 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.543113 2580 scope.go:117] "RemoveContainer" containerID="f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af" Sep 11 12:04:42.560732 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.560707 2580 scope.go:117] "RemoveContainer" containerID="f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af" Sep 11 12:04:42.561028 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:04:42.560994 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af\": container with ID starting with f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af not found: ID does not exist" containerID="f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af" Sep 11 12:04:42.561127 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.561059 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af"} err="failed to get container status \"f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af\": rpc error: code = NotFound desc = could not find container \"f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af\": container with ID starting with f696dc6507c0065b327870201afec49dd2e3a9b8be32e48b08a9021b4389c5af not found: ID does not exist" Sep 11 12:04:42.569806 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.569782 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6"] Sep 11 12:04:42.574795 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.574761 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-117711-6f6f768bb4-dnxs6"] Sep 11 12:04:42.872961 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:42.872881 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8c8284-f727-43ae-854e-c60c03b0438b" path="/var/lib/kubelet/pods/2a8c8284-f727-43ae-854e-c60c03b0438b/volumes" Sep 11 12:04:44.620814 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:44.620770 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:04:50.747052 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:50.747009 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f"] Sep 11 12:04:50.747536 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:50.747297 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" containerID="cri-o://dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5" gracePeriod=30 Sep 11 12:04:54.419939 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:54.419904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:04:56.001149 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.001123 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 12:04:56.120140 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120012 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-credential-socket\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120140 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120105 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-certs\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120148 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-podinfo\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-socket\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120208 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqml5\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-kube-api-access-lqml5\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120231 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-data\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120255 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-envoy\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120290 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-token\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120375 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120325 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:56.120727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120437 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:56.120727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120513 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1a741-2564-486d-910c-e2c33bf91e94-istiod-ca-cert\") pod \"e3f1a741-2564-486d-910c-e2c33bf91e94\" (UID: \"e3f1a741-2564-486d-910c-e2c33bf91e94\") " Sep 11 12:04:56.120727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120556 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:56.120727 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120586 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:56.120913 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120881 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.120997 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120935 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-workload-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.121131 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.120986 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f1a741-2564-486d-910c-e2c33bf91e94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:04:56.121217 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.121061 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-data\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.121217 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.121182 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-credential-socket\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.122624 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.122595 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:04:56.122723 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.122595 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-kube-api-access-lqml5" (OuterVolumeSpecName: "kube-api-access-lqml5") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "kube-api-access-lqml5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:04:56.122968 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.122937 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:04:56.122968 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.122937 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3f1a741-2564-486d-910c-e2c33bf91e94" (UID: "e3f1a741-2564-486d-910c-e2c33bf91e94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:04:56.221665 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.221628 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-token\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.221665 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.221660 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f1a741-2564-486d-910c-e2c33bf91e94-istiod-ca-cert\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.221665 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.221669 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-podinfo\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.221919 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.221679 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqml5\" (UniqueName: \"kubernetes.io/projected/e3f1a741-2564-486d-910c-e2c33bf91e94-kube-api-access-lqml5\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.221919 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.221689 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f1a741-2564-486d-910c-e2c33bf91e94-istio-envoy\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:04:56.629120 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.629088 2580 generic.go:356] "Generic (PLEG): container finished" podID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerID="dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5" exitCode=0 Sep 11 12:04:56.629301 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.629146 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" Sep 11 12:04:56.629301 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.629157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" event={"ID":"e3f1a741-2564-486d-910c-e2c33bf91e94","Type":"ContainerDied","Data":"dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5"} Sep 11 12:04:56.629301 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.629202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f" event={"ID":"e3f1a741-2564-486d-910c-e2c33bf91e94","Type":"ContainerDied","Data":"2f7f56e1ec2bac9a0992af9d0bb54ff8cb166359b43534dacee49b908ba5be0c"} Sep 11 12:04:56.629301 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.629224 2580 scope.go:117] "RemoveContainer" containerID="dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5" Sep 11 12:04:56.639430 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.639408 2580 scope.go:117] "RemoveContainer" containerID="1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748" Sep 11 12:04:56.648312 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.648294 2580 scope.go:117] "RemoveContainer" containerID="dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5" Sep 11 12:04:56.648614 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:04:56.648594 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5\": container with ID starting with dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5 not found: ID does not exist" containerID="dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5" Sep 11 12:04:56.648690 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.648623 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5"} err="failed to get container status \"dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5\": rpc error: code = NotFound desc = could not find container \"dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5\": container with ID starting with dddaeeb2a27e391c317f8c42218ab72874f12977d8d20f5cdc5808f726706ac5 not found: ID does not exist" Sep 11 12:04:56.648690 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.648642 2580 scope.go:117] "RemoveContainer" containerID="1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748" Sep 11 12:04:56.648902 ip-10-0-135-183 kubenswrapper[2580]: E0911 12:04:56.648886 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748\": container with ID starting with 1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748 not found: ID does not exist" containerID="1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748" Sep 11 12:04:56.648945 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.648909 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748"} err="failed to get container status \"1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748\": rpc error: code = NotFound desc = could not find container \"1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748\": container with ID starting with 1c6fe0b93be9b6fa179bb209e23911ae2952fb83b33a5bea4883b7ed57d77748 not found: ID does not exist" Sep 11 12:04:56.653916 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.653892 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f"] Sep 11 12:04:56.657841 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.657818 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-117711-openshift-default-77fc5586c4-4xz8f"] Sep 11 12:04:56.873163 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:04:56.873133 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" path="/var/lib/kubelet/pods/e3f1a741-2564-486d-910c-e2c33bf91e94/volumes" Sep 11 12:06:15.716523 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:06:15.716480 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:06:39.616692 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:06:39.616658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:06:47.415428 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:06:47.415393 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:06:55.699837 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:06:55.699754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:07:13.016311 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:07:13.016274 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:07:19.129950 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:07:19.129906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:08:45.114217 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:08:45.114179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:08:53.410534 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:08:53.410487 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:08:59.065385 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:08:59.065352 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 12:08:59.068801 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:08:59.068777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 12:09:01.211301 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:01.211265 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:09:14.215532 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:14.215494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:09:34.114176 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:34.114137 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:09:42.918755 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:42.918717 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:09:52.411264 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:52.411183 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:09:53.804526 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804490 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf"] Sep 11 12:09:53.804916 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804902 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9017fc15-d16a-4de7-9c0d-60ab92a66c4c" containerName="maas-api" Sep 11 12:09:53.804916 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804914 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9017fc15-d16a-4de7-9c0d-60ab92a66c4c" containerName="maas-api" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804923 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a982545-f86c-4ab1-8ad4-3c038321e8a9" containerName="main" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804929 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a982545-f86c-4ab1-8ad4-3c038321e8a9" containerName="main" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804941 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804948 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804955 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804960 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804974 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8c8284-f727-43ae-854e-c60c03b0438b" containerName="maas-api" Sep 11 12:09:53.804995 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.804979 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8c8284-f727-43ae-854e-c60c03b0438b" containerName="maas-api" Sep 11 12:09:53.805293 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.805048 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" Sep 11 12:09:53.805293 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.805058 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8c8284-f727-43ae-854e-c60c03b0438b" containerName="maas-api" Sep 11 12:09:53.805293 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.805065 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a982545-f86c-4ab1-8ad4-3c038321e8a9" containerName="main" Sep 11 12:09:53.805293 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.805071 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" Sep 11 12:09:53.805293 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.805077 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9017fc15-d16a-4de7-9c0d-60ab92a66c4c" containerName="maas-api" Sep 11 12:09:53.808325 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.808309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.812093 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.812068 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:09:53.812953 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.812934 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 12:09:53.813389 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.813373 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tztb5\"" Sep 11 12:09:53.813483 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.813439 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 12:09:53.813483 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.813450 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 12:09:53.835783 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.835751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf"] Sep 11 12:09:53.876340 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.876308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-maas-api-tls\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.876501 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.876359 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2hn\" (UniqueName: \"kubernetes.io/projected/ca85b309-43dc-48a4-bd2a-912970ae4d97-kube-api-access-9x2hn\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.876501 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.876439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca85b309-43dc-48a4-bd2a-912970ae4d97-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.876501 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.876492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-metrics-certs\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.977608 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.977568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-metrics-certs\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.977796 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.977636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-maas-api-tls\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.977796 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.977676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2hn\" (UniqueName: \"kubernetes.io/projected/ca85b309-43dc-48a4-bd2a-912970ae4d97-kube-api-access-9x2hn\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.977796 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.977710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca85b309-43dc-48a4-bd2a-912970ae4d97-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.978447 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.978391 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca85b309-43dc-48a4-bd2a-912970ae4d97-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.980349 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.980326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-maas-api-tls\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.980432 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.980326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-metrics-certs\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:53.990634 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:53.990606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2hn\" (UniqueName: \"kubernetes.io/projected/ca85b309-43dc-48a4-bd2a-912970ae4d97-kube-api-access-9x2hn\") pod \"maas-api-8f6fd9d5d-h6cpf\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:54.119309 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:54.119216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:54.412885 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:54.412854 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf"] Sep 11 12:09:54.413219 ip-10-0-135-183 kubenswrapper[2580]: W0911 12:09:54.413191 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca85b309_43dc_48a4_bd2a_912970ae4d97.slice/crio-f32273b95a48511fef51c86e6ab4e82ad71c22b2c774fd4a5c1f321d445c6b92 WatchSource:0}: Error finding container f32273b95a48511fef51c86e6ab4e82ad71c22b2c774fd4a5c1f321d445c6b92: Status 404 returned error can't find the container with id f32273b95a48511fef51c86e6ab4e82ad71c22b2c774fd4a5c1f321d445c6b92 Sep 11 12:09:54.414648 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:54.414632 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 12:09:55.289260 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:55.289225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" event={"ID":"ca85b309-43dc-48a4-bd2a-912970ae4d97","Type":"ContainerStarted","Data":"3941e9de8f6134ee130bdaa07c6ce58fbe2243c981462557ced7d1bed91aaa36"} Sep 11 12:09:55.289701 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:55.289289 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:09:55.289701 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:55.289313 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" event={"ID":"ca85b309-43dc-48a4-bd2a-912970ae4d97","Type":"ContainerStarted","Data":"f32273b95a48511fef51c86e6ab4e82ad71c22b2c774fd4a5c1f321d445c6b92"} Sep 11 12:09:55.316165 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:09:55.316111 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" podStartSLOduration=1.8327000020000002 podStartE2EDuration="2.316092602s" totalImagesPullingTime="483.3926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:09:55.313594165 +0000 UTC m=+2156.948187299" watchObservedRunningTime="2026-09-11 12:09:55.316092602 +0000 UTC m=+2156.950685736" Sep 11 12:10:01.151954 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.151918 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf"] Sep 11 12:10:01.152561 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.152200 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" podUID="ca85b309-43dc-48a4-bd2a-912970ae4d97" containerName="maas-api" containerID="cri-o://3941e9de8f6134ee130bdaa07c6ce58fbe2243c981462557ced7d1bed91aaa36" gracePeriod=30 Sep 11 12:10:01.158344 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.158315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:10:01.200081 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.200050 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-drx8g"] Sep 11 12:10:01.200439 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.200427 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" Sep 11 12:10:01.200486 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.200440 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" Sep 11 12:10:01.200486 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.200450 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" Sep 11 12:10:01.200486 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.200455 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" Sep 11 12:10:01.200591 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.200542 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f1a741-2564-486d-910c-e2c33bf91e94" containerName="istio-proxy" Sep 11 12:10:01.200591 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.200556 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e31339-c020-4ac0-a3b8-742ea9d01423" containerName="istio-proxy" Sep 11 12:10:01.203650 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.203628 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.217447 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.217416 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-drx8g"] Sep 11 12:10:01.327083 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.327024 2580 generic.go:356] "Generic (PLEG): container finished" podID="ca85b309-43dc-48a4-bd2a-912970ae4d97" containerID="3941e9de8f6134ee130bdaa07c6ce58fbe2243c981462557ced7d1bed91aaa36" exitCode=0 Sep 11 12:10:01.327373 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.327074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" event={"ID":"ca85b309-43dc-48a4-bd2a-912970ae4d97","Type":"ContainerDied","Data":"3941e9de8f6134ee130bdaa07c6ce58fbe2243c981462557ced7d1bed91aaa36"} Sep 11 12:10:01.348357 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.348308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ac188c-9658-497c-b266-3f1c61e09c8a-maas-api-tls\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.348552 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.348364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac188c-9658-497c-b266-3f1c61e09c8a-openshift-service-ca\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.348552 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.348446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ac188c-9658-497c-b266-3f1c61e09c8a-metrics-certs\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.348552 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.348482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lcz\" (UniqueName: \"kubernetes.io/projected/26ac188c-9658-497c-b266-3f1c61e09c8a-kube-api-access-r2lcz\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.408884 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.408819 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:10:01.449343 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.449308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ac188c-9658-497c-b266-3f1c61e09c8a-maas-api-tls\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.449343 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.449343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac188c-9658-497c-b266-3f1c61e09c8a-openshift-service-ca\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.449547 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.449368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ac188c-9658-497c-b266-3f1c61e09c8a-metrics-certs\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.449547 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.449392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lcz\" (UniqueName: \"kubernetes.io/projected/26ac188c-9658-497c-b266-3f1c61e09c8a-kube-api-access-r2lcz\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.450100 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.450021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ac188c-9658-497c-b266-3f1c61e09c8a-openshift-service-ca\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.452236 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.452210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ac188c-9658-497c-b266-3f1c61e09c8a-metrics-certs\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.452339 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.452278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26ac188c-9658-497c-b266-3f1c61e09c8a-maas-api-tls\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.460805 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.460778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lcz\" (UniqueName: \"kubernetes.io/projected/26ac188c-9658-497c-b266-3f1c61e09c8a-kube-api-access-r2lcz\") pod \"maas-api-796ffbb698-drx8g\" (UID: \"26ac188c-9658-497c-b266-3f1c61e09c8a\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.497988 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.497950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:10:01.548591 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.548558 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:01.549775 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.549753 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x2hn\" (UniqueName: \"kubernetes.io/projected/ca85b309-43dc-48a4-bd2a-912970ae4d97-kube-api-access-9x2hn\") pod \"ca85b309-43dc-48a4-bd2a-912970ae4d97\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " Sep 11 12:10:01.549848 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.549816 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca85b309-43dc-48a4-bd2a-912970ae4d97-openshift-service-ca\") pod \"ca85b309-43dc-48a4-bd2a-912970ae4d97\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " Sep 11 12:10:01.549886 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.549849 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-maas-api-tls\") pod \"ca85b309-43dc-48a4-bd2a-912970ae4d97\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " Sep 11 12:10:01.549886 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.549872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-metrics-certs\") pod \"ca85b309-43dc-48a4-bd2a-912970ae4d97\" (UID: \"ca85b309-43dc-48a4-bd2a-912970ae4d97\") " Sep 11 12:10:01.550239 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.550192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca85b309-43dc-48a4-bd2a-912970ae4d97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca85b309-43dc-48a4-bd2a-912970ae4d97" (UID: "ca85b309-43dc-48a4-bd2a-912970ae4d97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:10:01.552252 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.552229 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca85b309-43dc-48a4-bd2a-912970ae4d97-kube-api-access-9x2hn" (OuterVolumeSpecName: "kube-api-access-9x2hn") pod "ca85b309-43dc-48a4-bd2a-912970ae4d97" (UID: "ca85b309-43dc-48a4-bd2a-912970ae4d97"). InnerVolumeSpecName "kube-api-access-9x2hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:10:01.552359 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.552277 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca85b309-43dc-48a4-bd2a-912970ae4d97" (UID: "ca85b309-43dc-48a4-bd2a-912970ae4d97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:10:01.552359 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.552319 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca85b309-43dc-48a4-bd2a-912970ae4d97" (UID: "ca85b309-43dc-48a4-bd2a-912970ae4d97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:10:01.651018 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.650986 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-maas-api-tls\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:10:01.651018 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.651019 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca85b309-43dc-48a4-bd2a-912970ae4d97-metrics-certs\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:10:01.651249 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.651057 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x2hn\" (UniqueName: \"kubernetes.io/projected/ca85b309-43dc-48a4-bd2a-912970ae4d97-kube-api-access-9x2hn\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:10:01.651249 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.651076 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca85b309-43dc-48a4-bd2a-912970ae4d97-openshift-service-ca\") on node \"ip-10-0-135-183.ec2.internal\" DevicePath \"\"" Sep 11 12:10:01.827185 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:01.827158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-drx8g"] Sep 11 12:10:01.829903 ip-10-0-135-183 kubenswrapper[2580]: W0911 12:10:01.829878 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ac188c_9658_497c_b266_3f1c61e09c8a.slice/crio-a70c7b374a9cf60211af765c23d9ed767740098af1dc75c84990bf0708534b98 WatchSource:0}: Error finding container a70c7b374a9cf60211af765c23d9ed767740098af1dc75c84990bf0708534b98: Status 404 returned error can't find the container with id a70c7b374a9cf60211af765c23d9ed767740098af1dc75c84990bf0708534b98 Sep 11 12:10:02.336211 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:02.336169 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" event={"ID":"26ac188c-9658-497c-b266-3f1c61e09c8a","Type":"ContainerStarted","Data":"a70c7b374a9cf60211af765c23d9ed767740098af1dc75c84990bf0708534b98"} Sep 11 12:10:02.337531 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:02.337509 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" event={"ID":"ca85b309-43dc-48a4-bd2a-912970ae4d97","Type":"ContainerDied","Data":"f32273b95a48511fef51c86e6ab4e82ad71c22b2c774fd4a5c1f321d445c6b92"} Sep 11 12:10:02.337614 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:02.337556 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf" Sep 11 12:10:02.337654 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:02.337560 2580 scope.go:117] "RemoveContainer" containerID="3941e9de8f6134ee130bdaa07c6ce58fbe2243c981462557ced7d1bed91aaa36" Sep 11 12:10:02.363749 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:02.363717 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf"] Sep 11 12:10:02.368686 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:02.368657 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-h6cpf"] Sep 11 12:10:02.873150 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:02.873117 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca85b309-43dc-48a4-bd2a-912970ae4d97" path="/var/lib/kubelet/pods/ca85b309-43dc-48a4-bd2a-912970ae4d97/volumes" Sep 11 12:10:04.351228 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:04.351190 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" event={"ID":"26ac188c-9658-497c-b266-3f1c61e09c8a","Type":"ContainerStarted","Data":"cb82c95845c3c1410074efe3a94b389126bb537cc1c9f47af4c58473641c0784"} Sep 11 12:10:04.351641 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:04.351311 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:04.370830 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:04.370771 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" podStartSLOduration=1.418133141 podStartE2EDuration="3.370753333s" totalImagesPullingTime="1.952620192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:10:04.369633967 +0000 UTC m=+2166.004227100" watchObservedRunningTime="2026-09-11 12:10:04.370753333 +0000 UTC m=+2166.005346463" Sep 11 12:10:08.329692 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:08.329650 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:10:10.363116 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:10.363086 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796ffbb698-drx8g" Sep 11 12:10:16.199698 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:16.199662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:10:21.608203 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:21.608169 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8cltk"] Sep 11 12:10:29.534121 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:29.534079 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nd2wj_95b350c5-520e-4ea8-955f-d7fa2d83ae7f/manager/0.log" Sep 11 12:10:29.651424 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:29.651385 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-54qdh_b38de991-95e5-4937-8e7c-9f758d3508bf/manager/0.log" Sep 11 12:10:29.870420 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:29.870323 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4tsxp_6147e320-8c30-40a7-a4b1-a5b797d567a6/server/0.log" Sep 11 12:10:29.980086 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:29.980057 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-584lz_7877e7b4-0bd3-4672-8271-7a74f775da30/manager/0.log" Sep 11 12:10:30.392122 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:30.392096 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f769d976-vnk2p_9227851a-51c5-4558-9d9f-a28c5a792e70/copy-manifests/0.log" Sep 11 12:10:30.398882 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:30.398860 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f769d976-vnk2p_9227851a-51c5-4558-9d9f-a28c5a792e70/manager/0.log" Sep 11 12:10:30.727207 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:30.727169 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796ffbb698-drx8g_26ac188c-9658-497c-b266-3f1c61e09c8a/maas-api/0.log" Sep 11 12:10:31.045116 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:31.045000 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818800-mxzkv_7b69ae61-c5cd-42b8-bc22-f37b06f25f56/cleanup/0.log" Sep 11 12:10:32.248753 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:32.248723 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8jhws_6d223b6f-0d34-4b65-a2a3-e9a914399a3e/kuadrant-console-plugin/0.log" Sep 11 12:10:32.492318 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:32.492292 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8cltk_5034794a-d3e7-4a1f-b7a5-8f23f9817239/limitador/0.log" Sep 11 12:10:32.947446 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:32.947410 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw_5487c2f7-ffbd-4e85-a8eb-6feae50e4821/istio-proxy/0.log" Sep 11 12:10:33.404681 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:33.404658 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-l8td8_0233a32a-dd49-4369-974e-3e82d7103a23/istio-proxy/1.log" Sep 11 12:10:33.772519 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:33.772435 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fcb677dbb-2nz2z_064268d3-1d9f-48d1-8c10-da03ed41b29c/router/0.log" Sep 11 12:10:34.211863 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:34.211837 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-d692g_dce38404-0099-44ca-9f81-89b91ce2c160/main/0.log" Sep 11 12:10:34.438964 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:34.438936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-d8mwh_cd736a12-ce18-47ee-8634-c8f62988ecf9/main/0.log" Sep 11 12:10:34.770361 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:34.770336 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-ksxz7_dddc057a-c848-4d9f-9ee9-649f3a695755/main/0.log" Sep 11 12:10:43.712140 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:43.712107 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7nn5q_ebef619d-9d2e-4a27-9b1c-300daad95756/global-pull-secret-syncer/0.log" Sep 11 12:10:43.941299 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:43.941274 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cbw5p_1f853954-d580-4808-ad19-480303016858/konnectivity-agent/0.log" Sep 11 12:10:44.031525 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:44.031448 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-183.ec2.internal_93f10d8d545a409e6cc084480a99f98b/haproxy/0.log" Sep 11 12:10:48.662744 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:48.662715 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8jhws_6d223b6f-0d34-4b65-a2a3-e9a914399a3e/kuadrant-console-plugin/0.log" Sep 11 12:10:48.752713 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:48.752637 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8cltk_5034794a-d3e7-4a1f-b7a5-8f23f9817239/limitador/0.log" Sep 11 12:10:50.814286 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:50.814257 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-zld2w_06d8735e-012c-40ae-99db-84fba64b0bc2/cluster-monitoring-operator/0.log" Sep 11 12:10:51.184339 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:51.184314 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9zv9_d472299c-8964-4f5b-895a-53d87e2224a0/node-exporter/0.log" Sep 11 12:10:51.217006 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:51.216984 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9zv9_d472299c-8964-4f5b-895a-53d87e2224a0/kube-rbac-proxy/0.log" Sep 11 12:10:51.245080 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:51.245030 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9zv9_d472299c-8964-4f5b-895a-53d87e2224a0/init-textfile/0.log" Sep 11 12:10:51.892348 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:51.892266 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-vlj5n_5859eafb-5526-4708-8354-06c3f247d6f1/prometheus-operator-admission-webhook/0.log" Sep 11 12:10:53.690086 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:53.690055 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-7dsg8_fa5874ab-3b5c-4d44-8a50-ea744be7834d/networking-console-plugin/0.log" Sep 11 12:10:54.349138 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.349106 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/1.log" Sep 11 12:10:54.353516 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.353496 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-ldgwv_df17a3aa-2361-405d-8165-4f5dd30e4318/console-operator/2.log" Sep 11 12:10:54.513956 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.513928 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs"] Sep 11 12:10:54.514339 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.514326 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca85b309-43dc-48a4-bd2a-912970ae4d97" containerName="maas-api" Sep 11 12:10:54.514383 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.514341 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca85b309-43dc-48a4-bd2a-912970ae4d97" containerName="maas-api" Sep 11 12:10:54.514418 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.514403 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca85b309-43dc-48a4-bd2a-912970ae4d97" containerName="maas-api" Sep 11 12:10:54.517662 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.517646 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.532953 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.532930 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ntf2c\"/\"default-dockercfg-djvfd\"" Sep 11 12:10:54.533531 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.533518 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ntf2c\"/\"kube-root-ca.crt\"" Sep 11 12:10:54.533531 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.533527 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ntf2c\"/\"openshift-service-ca.crt\"" Sep 11 12:10:54.566595 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.566562 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs"] Sep 11 12:10:54.621069 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.620969 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-proc\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.621069 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.621001 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjn2\" (UniqueName: \"kubernetes.io/projected/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-kube-api-access-wgjn2\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.621069 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.621053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-podres\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.621317 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.621140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-lib-modules\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.621317 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.621178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-sys\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722609 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-sys\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-proc\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjn2\" (UniqueName: \"kubernetes.io/projected/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-kube-api-access-wgjn2\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-sys\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-podres\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722781 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-proc\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-lib-modules\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-podres\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.722952 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.722904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-lib-modules\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.731874 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.731854 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjn2\" (UniqueName: \"kubernetes.io/projected/d7a4fd57-0eb7-4550-93a1-8850ad89d3ea-kube-api-access-wgjn2\") pod \"perf-node-gather-daemonset-jkmvs\" (UID: \"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea\") " pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:54.827747 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:54.827721 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:55.110920 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:55.110885 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs"] Sep 11 12:10:55.112421 ip-10-0-135-183 kubenswrapper[2580]: W0911 12:10:55.112389 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd7a4fd57_0eb7_4550_93a1_8850ad89d3ea.slice/crio-a419a78104a4b988970bb105b2b4821514395dd6d26bbc174352ea1dbba10598 WatchSource:0}: Error finding container a419a78104a4b988970bb105b2b4821514395dd6d26bbc174352ea1dbba10598: Status 404 returned error can't find the container with id a419a78104a4b988970bb105b2b4821514395dd6d26bbc174352ea1dbba10598 Sep 11 12:10:55.121190 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:55.121171 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-qvl9x_3609dc30-fcab-4fb8-bc9a-033e50ef6a3e/download-server/0.log" Sep 11 12:10:55.632057 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:55.631998 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" event={"ID":"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea","Type":"ContainerStarted","Data":"acb435e4dd7c256d0c55ec1edc500f71fb3be636fb1e96c550115c4bdc573481"} Sep 11 12:10:55.632240 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:55.632066 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" event={"ID":"d7a4fd57-0eb7-4550-93a1-8850ad89d3ea","Type":"ContainerStarted","Data":"a419a78104a4b988970bb105b2b4821514395dd6d26bbc174352ea1dbba10598"} Sep 11 12:10:55.632240 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:55.632089 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:10:55.658061 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:55.657980 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" podStartSLOduration=1.6579658290000001 podStartE2EDuration="1.657965829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:10:55.656129647 +0000 UTC m=+2217.290722780" watchObservedRunningTime="2026-09-11 12:10:55.657965829 +0000 UTC m=+2217.292558979" Sep 11 12:10:55.814177 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:55.814149 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-4f8rc_a2713d0e-8325-4a04-920e-7ed39b1afb9e/volume-data-source-validator/0.log" Sep 11 12:10:56.945912 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:56.945884 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jd48q_40526ae2-253a-434c-bff9-5508117eb6c0/dns/0.log" Sep 11 12:10:56.971331 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:56.971310 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jd48q_40526ae2-253a-434c-bff9-5508117eb6c0/kube-rbac-proxy/0.log" Sep 11 12:10:57.160135 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:57.160109 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr9xg_543efb58-4920-4bba-bb4b-a3a4411fa7a4/dns-node-resolver/0.log" Sep 11 12:10:57.863555 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:57.863531 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vskzf_4ea77448-b67e-44e0-a935-9d6c03745028/node-ca/0.log" Sep 11 12:10:58.879627 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:58.879599 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw_5487c2f7-ffbd-4e85-a8eb-6feae50e4821/istio-proxy/0.log" Sep 11 12:10:59.197508 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:59.197477 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-l8td8_0233a32a-dd49-4369-974e-3e82d7103a23/istio-proxy/0.log" Sep 11 12:10:59.205703 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:59.205679 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-l8td8_0233a32a-dd49-4369-974e-3e82d7103a23/istio-proxy/1.log" Sep 11 12:10:59.314254 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:10:59.314232 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fcb677dbb-2nz2z_064268d3-1d9f-48d1-8c10-da03ed41b29c/router/0.log" Sep 11 12:11:00.291222 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:00.291195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2gldc_033fa8a3-2baf-401c-9792-759c21d55ea0/serve-healthcheck-canary/0.log" Sep 11 12:11:00.892603 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:00.892574 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-x7dfh_e3a4679a-bf25-4d80-810e-4df2cf36dd98/insights-operator/0.log" Sep 11 12:11:00.893112 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:00.893097 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-x7dfh_e3a4679a-bf25-4d80-810e-4df2cf36dd98/insights-operator/1.log" Sep 11 12:11:00.927722 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:00.927695 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6dx8n_ea9b246f-988f-4e91-8380-14ddd8a51ce4/kube-rbac-proxy/0.log" Sep 11 12:11:00.956322 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:00.956298 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6dx8n_ea9b246f-988f-4e91-8380-14ddd8a51ce4/exporter/0.log" Sep 11 12:11:00.980579 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:00.980555 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6dx8n_ea9b246f-988f-4e91-8380-14ddd8a51ce4/extractor/0.log" Sep 11 12:11:01.648067 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:01.648024 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ntf2c/perf-node-gather-daemonset-jkmvs" Sep 11 12:11:03.585826 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:03.581590 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nd2wj_95b350c5-520e-4ea8-955f-d7fa2d83ae7f/manager/0.log" Sep 11 12:11:03.762145 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:03.762117 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-54qdh_b38de991-95e5-4937-8e7c-9f758d3508bf/manager/0.log" Sep 11 12:11:03.853228 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:03.853201 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4tsxp_6147e320-8c30-40a7-a4b1-a5b797d567a6/server/0.log" Sep 11 12:11:03.938078 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:03.938000 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-584lz_7877e7b4-0bd3-4672-8271-7a74f775da30/manager/0.log" Sep 11 12:11:04.193892 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:04.193809 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f769d976-vnk2p_9227851a-51c5-4558-9d9f-a28c5a792e70/manager/0.log" Sep 11 12:11:04.232789 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:04.232766 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f769d976-vnk2p_9227851a-51c5-4558-9d9f-a28c5a792e70/copy-manifests/0.log" Sep 11 12:11:05.177508 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:05.177432 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw_5487c2f7-ffbd-4e85-a8eb-6feae50e4821/istio-proxy/0.log" Sep 11 12:11:05.492830 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:05.492718 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-l8td8_0233a32a-dd49-4369-974e-3e82d7103a23/istio-proxy/0.log" Sep 11 12:11:05.502280 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:05.502258 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-l8td8_0233a32a-dd49-4369-974e-3e82d7103a23/istio-proxy/1.log" Sep 11 12:11:05.596266 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:05.596231 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fcb677dbb-2nz2z_064268d3-1d9f-48d1-8c10-da03ed41b29c/router/0.log" Sep 11 12:11:05.617294 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:05.617273 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77956949cf-8bk7g_0efba69d-c832-4a4b-9980-68816f34af46/manager/0.log" Sep 11 12:11:07.129838 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:07.129807 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl5vgw_5487c2f7-ffbd-4e85-a8eb-6feae50e4821/istio-proxy/0.log" Sep 11 12:11:07.416382 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:07.416353 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-l8td8_0233a32a-dd49-4369-974e-3e82d7103a23/istio-proxy/0.log" Sep 11 12:11:07.424445 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:07.424419 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-l8td8_0233a32a-dd49-4369-974e-3e82d7103a23/istio-proxy/1.log" Sep 11 12:11:07.506704 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:07.506678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fcb677dbb-2nz2z_064268d3-1d9f-48d1-8c10-da03ed41b29c/router/0.log" Sep 11 12:11:11.271795 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:11.271763 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rpx62_fea7ab92-bea1-4209-bd83-490414b5ebcd/migrator/0.log" Sep 11 12:11:11.299015 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:11.298990 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rpx62_fea7ab92-bea1-4209-bd83-490414b5ebcd/graceful-termination/0.log" Sep 11 12:11:11.699258 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:11.699222 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-rxcpr_e5ecb8de-47a3-4c7a-90c8-099eb148851c/kube-storage-version-migrator-operator/1.log" Sep 11 12:11:11.700022 ip-10-0-135-183 kubenswrapper[2580]: I0911 12:11:11.700003 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-rxcpr_e5ecb8de-47a3-4c7a-90c8-099eb148851c/kube-storage-version-migrator-operator/0.log"