Sep 15 12:23:28.526217 ip-10-0-137-242 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 15 12:23:28.526232 ip-10-0-137-242 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 15 12:23:28.526243 ip-10-0-137-242 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 15 12:23:28.526684 ip-10-0-137-242 systemd[1]: Failed to start Kubernetes Kubelet. Sep 15 12:23:38.634895 ip-10-0-137-242 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 15 12:23:38.634912 ip-10-0-137-242 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot da67c1ea2a85490c99aa7bc79c0c2e81 -- Sep 15 12:25:18.537745 ip-10-0-137-242 systemd[1]: Starting Kubernetes Kubelet... Sep 15 12:25:18.949115 ip-10-0-137-242 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 12:25:18.949115 ip-10-0-137-242 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 12:25:18.949115 ip-10-0-137-242 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 12:25:18.949115 ip-10-0-137-242 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 12:25:18.949115 ip-10-0-137-242 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 12:25:18.951108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.950987 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 12:25:18.955664 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955639 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 12:25:18.955664 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955660 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 12:25:18.955664 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955665 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 12:25:18.955664 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955668 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 12:25:18.955664 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955671 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955674 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955678 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955681 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955684 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955687 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955690 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955693 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955696 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955699 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955702 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955704 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955707 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955710 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955713 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955715 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955718 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955721 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955729 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955732 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 12:25:18.955866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955735 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955738 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955741 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955743 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955746 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955748 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955751 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955754 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955756 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955759 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955761 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955764 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955766 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955770 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955773 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955776 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955778 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955781 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955783 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955786 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 12:25:18.956365 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955788 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955791 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955793 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955796 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955799 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955801 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955804 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955807 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955810 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955812 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955815 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955817 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955822 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955826 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955829 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955832 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955835 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955838 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955841 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 12:25:18.956906 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955844 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955848 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955851 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955855 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955858 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955861 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955865 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955870 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955873 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955876 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955878 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955881 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955884 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955887 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955890 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955893 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955895 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955898 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955900 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 12:25:18.957375 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955903 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955908 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955910 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955913 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955915 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955918 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955921 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955923 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955926 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955928 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955931 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955933 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955936 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955939 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955942 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955945 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.955947 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956681 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956687 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956691 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 12:25:18.957862 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956694 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956697 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956700 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956705 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956709 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956712 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956716 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956719 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956722 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956725 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956728 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956731 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956733 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956736 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956740 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956742 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956745 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956748 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956750 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 12:25:18.958355 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956753 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956756 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956758 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956761 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956763 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956766 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956769 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956771 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956773 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956776 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956778 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956781 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956787 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956791 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956794 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956798 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956801 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956804 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956807 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 12:25:18.958866 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956809 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956812 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956815 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956818 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956820 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956823 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956825 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956828 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956830 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956834 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956836 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956839 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956841 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956844 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956846 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956849 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956851 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956853 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956856 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956858 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 12:25:18.959334 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956861 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956864 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956866 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956868 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956871 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956875 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956878 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956880 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956883 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956886 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956888 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956891 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956893 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956896 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956900 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956902 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956906 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956909 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956912 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956914 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 12:25:18.959913 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956917 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956919 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956923 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956926 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956928 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956931 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956933 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956936 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956939 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956941 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956943 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956946 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956949 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956951 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956954 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956956 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956959 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.956962 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957362 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957367 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 12:25:18.960426 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957370 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957372 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957390 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957393 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957396 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957399 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957402 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957405 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957408 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957410 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957413 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957415 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957418 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957420 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957426 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957429 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957433 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957436 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957439 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957442 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 12:25:18.960923 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957452 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957455 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957458 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957460 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957463 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957466 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957468 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957471 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957473 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957477 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957480 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957483 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957485 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957488 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957491 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957494 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957496 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957499 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957501 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 12:25:18.961425 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957503 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957506 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957509 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957518 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957520 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957523 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957526 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957528 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957531 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957534 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957536 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957539 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957541 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957544 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957549 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957552 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957555 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957558 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957561 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957564 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 12:25:18.961946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957567 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957571 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957575 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957578 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957580 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957583 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957586 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957588 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957591 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957594 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957596 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957598 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957601 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957604 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957606 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957609 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957611 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957614 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957616 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957619 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 12:25:18.962444 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957621 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957624 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957627 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957629 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957632 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957634 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957637 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957639 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957642 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957646 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957649 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957651 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957654 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957657 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957662 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957665 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957668 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.957670 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957745 2550 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957801 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 12:25:18.962939 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957810 2550 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957815 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957819 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957823 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957827 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957832 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957835 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957838 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957842 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957845 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957849 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957852 2550 flags.go:64] FLAG: --cgroup-root="" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957855 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957858 2550 flags.go:64] FLAG: --client-ca-file="" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957861 2550 flags.go:64] FLAG: --cloud-provider="external" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957865 2550 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957869 2550 flags.go:64] FLAG: --cluster-domain="" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957872 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957875 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957878 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957882 2550 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957886 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957889 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 12:25:18.963439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957892 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957895 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957898 2550 flags.go:64] FLAG: --contention-profiling="false" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957902 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957906 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957909 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957912 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957916 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957919 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957922 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957927 2550 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957930 2550 flags.go:64] FLAG: --enable-server="true" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957933 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957938 2550 flags.go:64] FLAG: --event-burst="100" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957942 2550 flags.go:64] FLAG: --event-qps="50" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957944 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957948 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957951 2550 flags.go:64] FLAG: --eviction-hard="" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957954 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957958 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957961 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957964 2550 flags.go:64] FLAG: --eviction-soft="" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957967 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957969 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957973 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 12:25:18.963996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957976 2550 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957979 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957982 2550 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957984 2550 flags.go:64] FLAG: --feature-gates="" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957988 2550 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957991 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957994 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.957998 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958001 2550 flags.go:64] FLAG: --healthz-port="10248" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958004 2550 flags.go:64] FLAG: --help="false" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958007 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-137-242.ec2.internal" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958010 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958013 2550 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958016 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958019 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958023 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958026 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958031 2550 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958034 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958037 2550 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958040 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958043 2550 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958046 2550 flags.go:64] FLAG: --kube-reserved="" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958049 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 12:25:18.964643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958051 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958054 2550 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958057 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958060 2550 flags.go:64] FLAG: --lock-file="" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958063 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958066 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958069 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958074 2550 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958078 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958081 2550 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958084 2550 flags.go:64] FLAG: --logging-format="text" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958086 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958090 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958093 2550 flags.go:64] FLAG: --manifest-url="" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958095 2550 flags.go:64] FLAG: --manifest-url-header="" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958099 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958102 2550 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958107 2550 flags.go:64] FLAG: --max-pods="110" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958109 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958113 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958116 2550 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958119 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958122 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958125 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958127 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 12:25:18.965284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958137 2550 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958140 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958143 2550 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958146 2550 flags.go:64] FLAG: --pod-cidr="" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958149 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958154 2550 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958157 2550 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958160 2550 flags.go:64] FLAG: --pods-per-core="0" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958163 2550 flags.go:64] FLAG: --port="10250" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958166 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958169 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1ca3e2464bda704" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958172 2550 flags.go:64] FLAG: --qos-reserved="" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958175 2550 flags.go:64] FLAG: --read-only-port="10255" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958178 2550 flags.go:64] FLAG: --register-node="true" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958181 2550 flags.go:64] FLAG: --register-with-taints="" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958184 2550 flags.go:64] FLAG: --registry-burst="10" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958188 2550 flags.go:64] FLAG: --registry-qps="5" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958191 2550 flags.go:64] FLAG: --reserved-cpus="" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958194 2550 flags.go:64] FLAG: --reserved-memory="" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958198 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958201 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958204 2550 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958207 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958210 2550 flags.go:64] FLAG: --runonce="false" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958213 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 12:25:18.965943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958216 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958219 2550 flags.go:64] FLAG: --seccomp-default="false" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958222 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958225 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958227 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958231 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958234 2550 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958238 2550 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958241 2550 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958244 2550 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958247 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958250 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958253 2550 flags.go:64] FLAG: --system-cgroups="" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958256 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958261 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958264 2550 flags.go:64] FLAG: --tls-cert-file="" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958267 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958270 2550 flags.go:64] FLAG: --tls-min-version="" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958273 2550 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958276 2550 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958279 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958282 2550 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958285 2550 flags.go:64] FLAG: --v="2" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958290 2550 flags.go:64] FLAG: --version="false" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958295 2550 flags.go:64] FLAG: --vmodule="" Sep 15 12:25:18.966591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958299 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.958302 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958433 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958437 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958441 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958443 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958446 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958449 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958453 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958457 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958460 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958463 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958466 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958470 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958474 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958477 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958481 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958483 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958486 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 12:25:18.967182 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958489 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958491 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958494 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958497 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958500 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958502 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958505 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958507 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958510 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958513 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958516 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958521 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958524 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958527 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958531 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958533 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958536 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958539 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958541 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958544 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 12:25:18.967758 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958546 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958549 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958551 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958554 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958557 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958560 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958562 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958566 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958569 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958571 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958574 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958577 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958579 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958582 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958584 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958587 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958590 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958592 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958594 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958597 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 12:25:18.968246 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958599 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958602 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958605 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958609 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958611 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958614 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958618 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958620 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958623 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958625 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958628 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958631 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958633 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958636 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958639 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958641 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958644 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958647 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958650 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958654 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 12:25:18.968743 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958656 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958659 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958662 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958665 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958669 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958672 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958674 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958677 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958680 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958682 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958685 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958688 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958690 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958693 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958697 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958701 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958704 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958706 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958709 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 12:25:18.969247 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958712 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958715 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.958717 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.959542 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.967495 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.967513 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.967532 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.967538 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967594 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967601 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967605 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967609 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967612 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967615 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 12:25:18.969738 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967618 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967621 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967624 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967626 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967629 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967632 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967634 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967637 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967639 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967642 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967645 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967647 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967650 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967653 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967655 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967658 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967661 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967664 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967667 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967669 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 12:25:18.970095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967672 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967675 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967677 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967682 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967685 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967687 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967690 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967693 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967695 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967698 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967701 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967703 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967706 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967709 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967712 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967715 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967717 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967720 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967723 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967726 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 12:25:18.970607 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967728 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967731 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967734 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967737 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967739 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967742 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967744 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967747 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967750 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967753 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967755 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967758 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967761 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967764 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967766 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967769 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967772 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967775 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967778 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967780 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 12:25:18.971095 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967782 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967785 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967788 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967790 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967793 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967796 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967799 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967801 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967804 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967806 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967809 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967812 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967814 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967817 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967820 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967822 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967826 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967830 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967833 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967836 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 12:25:18.971601 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967838 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967841 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967843 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967846 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967849 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967851 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967854 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967856 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967859 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967862 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967865 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967869 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.967872 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.967877 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 12:25:18.972081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968005 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968010 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968014 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968016 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968019 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968022 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968024 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968027 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968030 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968032 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968035 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968038 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968040 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968043 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968046 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968048 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968051 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968053 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968056 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968059 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 12:25:18.972431 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968061 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968064 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968066 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968069 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968071 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968074 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968077 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968079 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968082 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968085 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968088 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968091 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968093 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968096 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968098 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968101 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968103 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968106 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968110 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968113 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 12:25:18.972924 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968116 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968119 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968122 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968126 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968129 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968132 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968135 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968138 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968141 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968144 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968148 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968150 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968153 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968156 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968158 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968161 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968164 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968166 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968168 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 12:25:18.973501 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968171 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968173 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968176 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968179 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968182 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968184 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968187 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968189 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968193 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968197 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968199 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968202 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968205 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968207 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968210 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968212 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968215 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968217 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968220 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 12:25:18.973963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968223 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968225 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968228 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968231 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968233 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968236 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968239 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968241 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968243 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968246 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968249 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968252 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968254 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968257 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968260 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968262 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968265 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968268 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968270 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968273 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 12:25:18.974455 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:18.968275 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 12:25:18.974937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.968281 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 12:25:18.974937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.968415 2550 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 12:25:18.974937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.971722 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 12:25:18.974937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.972856 2550 server.go:1019] "Starting client certificate rotation" Sep 15 12:25:18.974937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.972961 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 12:25:18.974937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.972990 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 12:25:18.975855 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.975841 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 12:25:18.977277 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.977258 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 12:25:18.994575 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.994424 2550 log.go:25] "Validated CRI v1 runtime API" Sep 15 12:25:18.998573 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.998552 2550 log.go:25] "Validated CRI v1 image API" Sep 15 12:25:18.999462 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:18.999440 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 12:25:19.002728 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.002706 2550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9a6d210a-ba9c-4aa6-a8d2-c302f0bb13b6:/dev/nvme0n1p4 fdc73245-f9a3-446e-927e-f35b8c340780:/dev/nvme0n1p3] Sep 15 12:25:19.002806 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.002726 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 12:25:19.005740 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.005721 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 12:25:19.008791 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.008669 2550 manager.go:216] Machine: {Timestamp:2026-09-15 12:25:19.007217147 +0000 UTC m=+0.362063461 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100831 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c86b05fc065a5ace7b065daeac838 SystemUUID:ec2c86b0-5fc0-65a5-ace7-b065daeac838 BootID:da67c1ea-2a85-490c-99aa-7bc79c0c2e81 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:a8:f8:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:a8:f8:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:c7:c3:c9:33:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 12:25:19.008791 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.008788 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 12:25:19.008925 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.008913 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 12:25:19.012425 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.012389 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 12:25:19.012602 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.012427 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 12:25:19.013450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.013431 2550 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 12:25:19.013520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.013459 2550 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 12:25:19.013520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.013471 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 12:25:19.015058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.015044 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 12:25:19.016304 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.016292 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 12:25:19.021471 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.021455 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 15 12:25:19.021793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.021779 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 12:25:19.021913 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.021822 2550 kubelet.go:486] "Attempting to sync node with API server" Sep 15 12:25:19.021913 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.021838 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 12:25:19.021913 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.021853 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 12:25:19.021913 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.021866 2550 kubelet.go:398] "Adding apiserver pod source" Sep 15 12:25:19.021913 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.021882 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 12:25:19.022288 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.022273 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 12:25:19.022353 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.022296 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 12:25:19.024699 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.024685 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 12:25:19.027744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.027729 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 12:25:19.027799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.027760 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 12:25:19.029771 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029756 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 12:25:19.029771 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029777 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029788 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029797 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029808 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029816 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029835 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029843 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029853 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029861 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029880 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 12:25:19.029909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.029890 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 12:25:19.030846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.030834 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 12:25:19.030901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.030848 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 12:25:19.032754 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.032731 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 12:25:19.032754 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.032730 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 12:25:19.035021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.035004 2550 server.go:1268] "Started kubelet" Sep 15 12:25:19.035110 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.035091 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 12:25:19.035211 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.035173 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 12:25:19.035267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.035231 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 12:25:19.036685 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.036654 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 12:25:19.036704 ip-10-0-137-242 systemd[1]: Started Kubernetes Kubelet. Sep 15 12:25:19.038550 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.038534 2550 server.go:310] "Adding debug handlers to kubelet server" Sep 15 12:25:19.040588 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.040544 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 12:25:19.043567 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.043550 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 12:25:19.044739 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.043599 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18d57e23c47aca0d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-09-15 12:25:19.034976781 +0000 UTC m=+0.389823095,LastTimestamp:2026-09-15 12:25:19.034976781 +0000 UTC m=+0.389823095,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Sep 15 12:25:19.045044 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.045028 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 12:25:19.045101 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.045049 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 12:25:19.045187 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.045159 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.046045 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.045937 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 12:25:19.046045 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.045954 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 12:25:19.046197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046055 2550 reconstruct.go:97] "Volume reconstruction finished" Sep 15 12:25:19.046197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046065 2550 reconciler.go:29] "Reconciler: start to sync state" Sep 15 12:25:19.046197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046131 2550 factory.go:55] Registering systemd factory Sep 15 12:25:19.046197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046146 2550 factory.go:223] Registration of the systemd container factory successfully Sep 15 12:25:19.046366 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046209 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 12:25:19.046466 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046452 2550 factory.go:153] Registering CRI-O factory Sep 15 12:25:19.046516 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046470 2550 factory.go:223] Registration of the crio container factory successfully Sep 15 12:25:19.046568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046523 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 12:25:19.046568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046542 2550 factory.go:103] Registering Raw factory Sep 15 12:25:19.046568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046557 2550 manager.go:1196] Started watching for new ooms in manager Sep 15 12:25:19.047038 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.046961 2550 manager.go:349] Starting recovery of all containers Sep 15 12:25:19.049133 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.049096 2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 15 12:25:19.049212 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.049197 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 15 12:25:19.054474 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.054447 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dcj44" Sep 15 12:25:19.058579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.058563 2550 manager.go:354] Recovery completed Sep 15 12:25:19.061890 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.061869 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dcj44" Sep 15 12:25:19.063506 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.063491 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 12:25:19.066732 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.066709 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Sep 15 12:25:19.066807 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.066749 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 12:25:19.066807 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.066764 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Sep 15 12:25:19.067368 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.067353 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 12:25:19.067472 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.067414 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 12:25:19.067472 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.067425 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 12:25:19.067472 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.067444 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 15 12:25:19.068075 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.068049 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Sep 15 12:25:19.068141 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.068078 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 12:25:19.068141 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.068091 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Sep 15 12:25:19.068743 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.068650 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-242.ec2.internal.18d57e23c65f51f5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-242.ec2.internal,UID:ip-10-0-137-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-242.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-242.ec2.internal,},FirstTimestamp:2026-09-15 12:25:19.066730997 +0000 UTC m=+0.421577311,LastTimestamp:2026-09-15 12:25:19.066730997 +0000 UTC m=+0.421577311,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-242.ec2.internal,}" Sep 15 12:25:19.069481 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.069468 2550 policy_none.go:49] "None policy: Start" Sep 15 12:25:19.069538 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.069485 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 12:25:19.069538 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.069496 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 12:25:19.071407 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.071397 2550 policy_none.go:47] "Start" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.107025 2550 manager.go:337] "Starting Device Plugin manager" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.107072 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.107086 2550 server.go:87] "Starting device plugin registration server" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.107370 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.107398 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.107570 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.107659 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.107670 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.108224 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 12:25:19.119527 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.108266 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.145476 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.145431 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 12:25:19.146717 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.146701 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 12:25:19.146787 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.146754 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 12:25:19.146833 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.146789 2550 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 12:25:19.146877 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.146836 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 12:25:19.149364 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.149339 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 12:25:19.208567 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.208461 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 12:25:19.209532 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.209509 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Sep 15 12:25:19.209644 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.209545 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 12:25:19.209644 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.209556 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Sep 15 12:25:19.209644 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.209586 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.219113 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.219095 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.219178 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.219119 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-242.ec2.internal\": node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.241964 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.241939 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.247129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.247108 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal"] Sep 15 12:25:19.247183 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.247179 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 12:25:19.248829 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.248809 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Sep 15 12:25:19.248894 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.248851 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 12:25:19.248894 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.248868 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Sep 15 12:25:19.251240 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.251227 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 12:25:19.251358 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.251343 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.251427 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.251375 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 12:25:19.251978 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.251963 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Sep 15 12:25:19.252047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.251987 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Sep 15 12:25:19.252047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.252020 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 12:25:19.252047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.252030 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Sep 15 12:25:19.252047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.251992 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 12:25:19.252170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.252063 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Sep 15 12:25:19.254273 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.254260 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.254326 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.254286 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 12:25:19.254963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.254927 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientMemory" Sep 15 12:25:19.254963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.254956 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 12:25:19.254963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.254967 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeHasSufficientPID" Sep 15 12:25:19.275031 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.275006 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-242.ec2.internal\" not found" node="ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.279547 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.279527 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-242.ec2.internal\" not found" node="ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.342545 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.342508 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.347229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.347202 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab53c1e711a5972e218bac8960973b27-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ab53c1e711a5972e218bac8960973b27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.347365 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.347234 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab53c1e711a5972e218bac8960973b27-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ab53c1e711a5972e218bac8960973b27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.347365 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.347253 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f58eb2157143886588c630e14dfef629-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"f58eb2157143886588c630e14dfef629\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.443120 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.443088 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.447369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.447351 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f58eb2157143886588c630e14dfef629-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"f58eb2157143886588c630e14dfef629\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.447468 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.447391 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab53c1e711a5972e218bac8960973b27-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ab53c1e711a5972e218bac8960973b27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.447468 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.447422 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab53c1e711a5972e218bac8960973b27-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ab53c1e711a5972e218bac8960973b27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.447468 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.447462 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab53c1e711a5972e218bac8960973b27-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ab53c1e711a5972e218bac8960973b27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.447576 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.447466 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ab53c1e711a5972e218bac8960973b27-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal\" (UID: \"ab53c1e711a5972e218bac8960973b27\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.447576 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.447470 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f58eb2157143886588c630e14dfef629-config\") pod \"kube-apiserver-proxy-ip-10-0-137-242.ec2.internal\" (UID: \"f58eb2157143886588c630e14dfef629\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.543837 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.543759 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.578269 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.578233 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.583294 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.583275 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Sep 15 12:25:19.644491 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.644441 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.750819 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.750779 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.850947 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.850850 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.951266 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:19.951223 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:19.973060 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.973020 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 12:25:19.973215 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:19.973193 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 12:25:20.051341 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:20.051313 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:20.067761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.067717 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 12:20:19 +0000 UTC" deadline="2028-03-18 20:16:10.878730602 +0000 UTC" Sep 15 12:25:20.067761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.067753 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13207h50m50.810983276s" Sep 15 12:25:20.152329 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:20.152253 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:20.216521 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.216481 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 12:25:20.252947 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:20.252908 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:20.353639 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:20.353596 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:20.454649 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:20.454413 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-242.ec2.internal\" not found" Sep 15 12:25:20.484555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.484527 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 12:25:20.491249 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.491221 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 12:25:20.514946 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:20.514909 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58eb2157143886588c630e14dfef629.slice/crio-ee33ff515171746120f509e776cd326a6a0b80bb234a740a1b4e5fc528644c5d WatchSource:0}: Error finding container ee33ff515171746120f509e776cd326a6a0b80bb234a740a1b4e5fc528644c5d: Status 404 returned error can't find the container with id ee33ff515171746120f509e776cd326a6a0b80bb234a740a1b4e5fc528644c5d Sep 15 12:25:20.523776 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.523755 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:25:20.527368 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:20.527342 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab53c1e711a5972e218bac8960973b27.slice/crio-bd12c4a2c874206cd7d0dd9d3f1bf8569e479eff7326bddc7fb122b5819c47c4 WatchSource:0}: Error finding container bd12c4a2c874206cd7d0dd9d3f1bf8569e479eff7326bddc7fb122b5819c47c4: Status 404 returned error can't find the container with id bd12c4a2c874206cd7d0dd9d3f1bf8569e479eff7326bddc7fb122b5819c47c4 Sep 15 12:25:20.545735 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.545704 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" Sep 15 12:25:20.561601 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.561578 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 12:25:20.562522 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.562507 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" Sep 15 12:25:20.571107 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:20.571090 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 12:25:21.026263 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.026210 2550 apiserver.go:52] "Watching apiserver" Sep 15 12:25:21.033243 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.033221 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 12:25:21.034370 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.034343 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-px8td","kube-system/konnectivity-agent-qgfwl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9","openshift-cluster-node-tuning-operator/tuned-68hjm","openshift-dns/node-resolver-fxrzq","openshift-multus/multus-w72xx","openshift-multus/network-metrics-daemon-bw2fs","openshift-network-operator/iptables-alerter-x2hbm","kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal","openshift-image-registry/node-ca-lnvms","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal","openshift-multus/multus-additional-cni-plugins-jdtrs","openshift-network-diagnostics/network-check-target-dr79g","openshift-ovn-kubernetes/ovnkube-node-99dbz"] Sep 15 12:25:21.037389 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.037367 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:21.037491 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.037456 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:21.041982 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.041953 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:21.042105 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.041956 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.042105 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.042059 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:21.044502 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.044353 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 12:25:21.044854 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.044776 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 12:25:21.044854 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.044793 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqcf2\"" Sep 15 12:25:21.044999 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.044841 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 12:25:21.046094 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.046070 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 12:25:21.046374 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.046351 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.048590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.048559 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x9pmc\"" Sep 15 12:25:21.048695 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.048593 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:25:21.048695 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.048599 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 12:25:21.048876 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.048834 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.050761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.050740 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 12:25:21.050919 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.050896 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 12:25:21.051150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.051130 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.051240 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.051198 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzzm6\"" Sep 15 12:25:21.053276 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.053255 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pwcpq\"" Sep 15 12:25:21.053384 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.053367 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 12:25:21.053478 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.053451 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 12:25:21.053535 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.053486 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 12:25:21.053588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.053571 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 12:25:21.053733 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.053716 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.055673 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.055654 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 12:25:21.055815 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.055800 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 12:25:21.055873 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.055826 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 12:25:21.056199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.056181 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.056276 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.056238 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:21.056912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.056893 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 12:25:21.057005 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.056922 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 12:25:21.057005 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.056956 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 12:25:21.057005 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.056893 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jx5lj\"" Sep 15 12:25:21.057174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.057007 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 12:25:21.058588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058564 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-tmp-dir\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.058679 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058601 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-device-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.058679 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058628 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysctl-d\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.058792 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058688 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmrs\" (UniqueName: \"kubernetes.io/projected/f4a9b48e-794e-41c2-920e-2ff5156631da-kube-api-access-hfmrs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:21.058792 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058718 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vb4\" (UniqueName: \"kubernetes.io/projected/363be6bc-33bd-4256-92e9-d8c4b3ad4787-kube-api-access-z2vb4\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.058792 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058757 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-socket-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.058792 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058783 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-kubernetes\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058804 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058809 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysctl-conf\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058861 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-lib-modules\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058878 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1580dcd6-b90a-491e-a1dc-f4cae36584a6-tmp\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058892 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-host\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058912 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058940 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-registration-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.059000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.058980 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-var-lib-kubelet\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059032 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-systemd\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059060 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-run\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059084 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-tuned\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059102 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059118 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lz6v\" (UniqueName: \"kubernetes.io/projected/1580dcd6-b90a-491e-a1dc-f4cae36584a6-kube-api-access-4lz6v\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059134 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-hosts-file\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059155 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6c9\" (UniqueName: \"kubernetes.io/projected/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-kube-api-access-qq6c9\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059179 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059199 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059213 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-sys-fs\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059233 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysconfig\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059260 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-modprobe-d\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.059406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.059292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-sys\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.060869 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.060849 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hkhnq\"" Sep 15 12:25:21.060988 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.060971 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 12:25:21.061054 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.061004 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 12:25:21.061275 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.061259 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.063582 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.063565 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 12:25:21.063878 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.063858 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 12:25:21.063993 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.063950 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.064125 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.064090 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 12:25:21.064403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.064374 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fwljz\"" Sep 15 12:25:21.066406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.066386 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.066843 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.066820 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:25:21.067166 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.067148 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 12:25:21.067253 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.067198 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v52jg\"" Sep 15 12:25:21.067253 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.067212 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 12:25:21.068702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.068684 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2bb7x\"" Sep 15 12:25:21.068806 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.068700 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 12:25:21.068806 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.068799 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 12:25:21.075837 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.075818 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n6vsj" Sep 15 12:25:21.088095 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.088073 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n6vsj" Sep 15 12:25:21.147474 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.147419 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 12:25:21.151266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.151219 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"f58eb2157143886588c630e14dfef629","Type":"ContainerStarted","Data":"ee33ff515171746120f509e776cd326a6a0b80bb234a740a1b4e5fc528644c5d"} Sep 15 12:25:21.152802 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.152771 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"ab53c1e711a5972e218bac8960973b27","Type":"ContainerStarted","Data":"bd12c4a2c874206cd7d0dd9d3f1bf8569e479eff7326bddc7fb122b5819c47c4"} Sep 15 12:25:21.160438 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160401 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-device-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.160438 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160439 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-slash\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160473 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovnkube-script-lib\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160490 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99vrc\" (UniqueName: \"kubernetes.io/projected/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-kube-api-access-99vrc\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160514 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-socket-dir-parent\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160546 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-device-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160552 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-netns\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160599 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-kubernetes\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160623 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysctl-conf\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160641 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:21.160666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160657 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-var-lib-kubelet\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160675 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160701 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09c54c4c-17f5-47a7-8d09-9f185f561093-konnectivity-ca\") pod \"konnectivity-agent-qgfwl\" (UID: \"09c54c4c-17f5-47a7-8d09-9f185f561093\") " pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160720 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-run\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160781 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-var-lib-kubelet\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160797 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-tuned\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160782 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-kubernetes\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.160805 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160849 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-run\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160843 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-var-lib-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160885 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovnkube-config\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.160937 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs podName:f4a9b48e-794e-41c2-920e-2ff5156631da nodeName:}" failed. No retries permitted until 2026-09-15 12:25:21.660887484 +0000 UTC m=+3.015733793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs") pod "network-metrics-daemon-bw2fs" (UID: "f4a9b48e-794e-41c2-920e-2ff5156631da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.160978 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161008 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-log-socket\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161037 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.161108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161061 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lz6v\" (UniqueName: \"kubernetes.io/projected/1580dcd6-b90a-491e-a1dc-f4cae36584a6-kube-api-access-4lz6v\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161130 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161160 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-run-netns\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161189 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-node-log\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161207 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqcs\" (UniqueName: \"kubernetes.io/projected/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-kube-api-access-hpqcs\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161234 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlhl\" (UniqueName: \"kubernetes.io/projected/337f5c5a-79e7-4bb1-b716-c397d3329687-kube-api-access-bzlhl\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161264 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysctl-conf\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161269 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161314 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161338 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-sys-fs\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161364 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysconfig\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161388 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-kubelet\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161422 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-etc-selinux\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161428 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-sys-fs\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161434 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161437 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysconfig\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161465 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/337f5c5a-79e7-4bb1-b716-c397d3329687-cni-binary-copy\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.161912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161502 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161522 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-sys\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161564 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-serviceca\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161594 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161605 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-sys\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161622 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-daemon-config\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161653 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-tmp-dir\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161679 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysctl-d\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161706 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-systemd\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161732 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-cni-multus\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161757 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8d9432-e366-40c2-81fd-b478d05a1df7-dbus\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161787 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmrs\" (UniqueName: \"kubernetes.io/projected/f4a9b48e-794e-41c2-920e-2ff5156631da-kube-api-access-hfmrs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161816 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vb4\" (UniqueName: \"kubernetes.io/projected/363be6bc-33bd-4256-92e9-d8c4b3ad4787-kube-api-access-z2vb4\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161843 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-system-cni-dir\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161846 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-sysctl-d\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161888 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-cni-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161920 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09c54c4c-17f5-47a7-8d09-9f185f561093-agent-certs\") pod \"konnectivity-agent-qgfwl\" (UID: \"09c54c4c-17f5-47a7-8d09-9f185f561093\") " pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.162723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161951 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-socket-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161967 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-tmp-dir\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.161986 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-lib-modules\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162018 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1580dcd6-b90a-491e-a1dc-f4cae36584a6-tmp\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162040 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pr7\" (UniqueName: \"kubernetes.io/projected/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-kube-api-access-52pr7\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162058 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-hostroot\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162078 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-multus-certs\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162088 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-socket-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162089 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-lib-modules\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162107 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-etc-kubernetes\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162169 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-host\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-registration-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162256 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-systemd-units\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162260 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-host\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162319 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/363be6bc-33bd-4256-92e9-d8c4b3ad4787-registration-dir\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162362 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-ovn\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.163379 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162398 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-cni-bin\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162423 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-cnibin\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162442 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-systemd\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162461 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovn-node-metrics-cert\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162498 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-k8s-cni-cncf-io\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162526 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-env-overrides\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162538 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-systemd\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162554 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-iptables-alerter-script\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162585 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tbp\" (UniqueName: \"kubernetes.io/projected/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-kube-api-access-m7tbp\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162616 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-etc-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162642 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162657 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-kubelet\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162676 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-hosts-file\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162703 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6c9\" (UniqueName: \"kubernetes.io/projected/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-kube-api-access-qq6c9\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162729 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-host\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162752 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-hosts-file\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162757 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-host-slash\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.164000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162791 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162819 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cnibin\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162848 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-system-cni-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162889 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-conf-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162926 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-modprobe-d\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162953 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-cni-netd\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.162975 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cni-binary-copy\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.163011 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-cni-bin\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.163024 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-modprobe-d\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.163048 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8d9432-e366-40c2-81fd-b478d05a1df7-kubelet-config\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.165592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.165570 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1580dcd6-b90a-491e-a1dc-f4cae36584a6-etc-tuned\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.165687 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.165594 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1580dcd6-b90a-491e-a1dc-f4cae36584a6-tmp\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.169980 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.169942 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 12:25:21.169980 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.169970 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 12:25:21.170167 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.169985 2550 projected.go:196] Error preparing data for projected volume kube-api-access-zf8g2 for pod openshift-network-diagnostics/network-check-target-dr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:21.170167 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.170068 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2 podName:e1339667-307a-4bfd-a75c-a9d3e8d81e2d nodeName:}" failed. No retries permitted until 2026-09-15 12:25:21.670047659 +0000 UTC m=+3.024893981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zf8g2" (UniqueName: "kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2") pod "network-check-target-dr79g" (UID: "e1339667-307a-4bfd-a75c-a9d3e8d81e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:21.172997 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.172974 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lz6v\" (UniqueName: \"kubernetes.io/projected/1580dcd6-b90a-491e-a1dc-f4cae36584a6-kube-api-access-4lz6v\") pod \"tuned-68hjm\" (UID: \"1580dcd6-b90a-491e-a1dc-f4cae36584a6\") " pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.172997 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.172986 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmrs\" (UniqueName: \"kubernetes.io/projected/f4a9b48e-794e-41c2-920e-2ff5156631da-kube-api-access-hfmrs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:21.173232 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.173063 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6c9\" (UniqueName: \"kubernetes.io/projected/d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55-kube-api-access-qq6c9\") pod \"node-resolver-fxrzq\" (UID: \"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55\") " pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.173990 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.173968 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vb4\" (UniqueName: \"kubernetes.io/projected/363be6bc-33bd-4256-92e9-d8c4b3ad4787-kube-api-access-z2vb4\") pod \"aws-ebs-csi-driver-node-t7lr9\" (UID: \"363be6bc-33bd-4256-92e9-d8c4b3ad4787\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.264344 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264300 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-var-lib-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264360 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovnkube-config\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264387 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.264533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264414 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-log-socket\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264450 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-run-netns\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264468 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-node-log\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264496 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqcs\" (UniqueName: \"kubernetes.io/projected/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-kube-api-access-hpqcs\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.264533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264532 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlhl\" (UniqueName: \"kubernetes.io/projected/337f5c5a-79e7-4bb1-b716-c397d3329687-kube-api-access-bzlhl\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.264750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264571 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-kubelet\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264595 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.264750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264615 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/337f5c5a-79e7-4bb1-b716-c397d3329687-cni-binary-copy\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.264750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-serviceca\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.264750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264671 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264693 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-daemon-config\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.264750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264715 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-systemd\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264756 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-cni-multus\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264777 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8d9432-e366-40c2-81fd-b478d05a1df7-dbus\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264801 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-system-cni-dir\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264821 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-cni-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264857 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09c54c4c-17f5-47a7-8d09-9f185f561093-agent-certs\") pod \"konnectivity-agent-qgfwl\" (UID: \"09c54c4c-17f5-47a7-8d09-9f185f561093\") " pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264881 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pr7\" (UniqueName: \"kubernetes.io/projected/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-kube-api-access-52pr7\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264901 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-hostroot\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264929 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-multus-certs\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.264963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264949 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-etc-kubernetes\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264970 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.264993 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-systemd-units\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265024 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-ovn\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265042 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-cni-bin\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265064 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-cnibin\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265086 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovn-node-metrics-cert\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265117 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-k8s-cni-cncf-io\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265139 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-env-overrides\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265164 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-iptables-alerter-script\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265194 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tbp\" (UniqueName: \"kubernetes.io/projected/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-kube-api-access-m7tbp\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265212 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-etc-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265234 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265255 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-kubelet\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265285 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-host\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265308 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-host-slash\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265326 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265349 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cnibin\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265376 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-system-cni-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265398 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-conf-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-cni-netd\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265439 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cni-binary-copy\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265479 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-cni-bin\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265502 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8d9432-e366-40c2-81fd-b478d05a1df7-kubelet-config\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265528 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-slash\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265551 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovnkube-script-lib\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265583 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99vrc\" (UniqueName: \"kubernetes.io/projected/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-kube-api-access-99vrc\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265606 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-socket-dir-parent\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265636 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-netns\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.265744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265672 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.266375 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.265694 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09c54c4c-17f5-47a7-8d09-9f185f561093-konnectivity-ca\") pod \"konnectivity-agent-qgfwl\" (UID: \"09c54c4c-17f5-47a7-8d09-9f185f561093\") " pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.266375 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.266311 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/09c54c4c-17f5-47a7-8d09-9f185f561093-konnectivity-ca\") pod \"konnectivity-agent-qgfwl\" (UID: \"09c54c4c-17f5-47a7-8d09-9f185f561093\") " pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.266473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.266414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-var-lib-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.266858 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.266840 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovnkube-config\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267066 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-log-socket\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267153 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267076 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-cni-multus\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.267153 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267132 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-cni-bin\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-run-netns\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267226 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.267267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267243 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-node-log\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267251 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267370 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267306 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-k8s-cni-cncf-io\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.267424 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267410 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-system-cni-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.267484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267472 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-conf-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.267525 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267508 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-cni-netd\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267787 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267762 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-env-overrides\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.267916 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267896 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-daemon-config\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.267982 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267910 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cni-binary-copy\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.268038 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.267987 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cnibin\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.268038 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268031 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-cni-bin\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.268138 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268077 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-systemd\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.268138 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268098 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-iptables-alerter-script\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.268242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268162 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-slash\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.268366 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268343 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-serviceca\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.268456 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268433 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-netns\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.268592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268503 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-etc-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.268592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268558 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-socket-dir-parent\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.268592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268571 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-var-lib-kubelet\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.268785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268584 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-host-run-multus-certs\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.268785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268632 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovnkube-script-lib\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.268785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268644 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-host\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.268785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268680 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-openvswitch\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.268785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268700 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-host-slash\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.268785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268762 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.269072 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268790 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.269072 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268821 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea8d9432-e366-40c2-81fd-b478d05a1df7-kubelet-config\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.269072 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.268939 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:21.269072 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.268994 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea8d9432-e366-40c2-81fd-b478d05a1df7-dbus\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.269072 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269001 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-system-cni-dir\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.269072 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.269018 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret podName:ea8d9432-e366-40c2-81fd-b478d05a1df7 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:21.768989965 +0000 UTC m=+3.123836267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret") pod "global-pull-secret-syncer-px8td" (UID: "ea8d9432-e366-40c2-81fd-b478d05a1df7") : object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:21.269395 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269084 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-systemd-units\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.269395 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269349 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-hostroot\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.269395 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269346 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/337f5c5a-79e7-4bb1-b716-c397d3329687-cni-binary-copy\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.269395 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269387 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-cnibin\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.269629 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269449 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-multus-cni-dir\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.269629 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269450 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/337f5c5a-79e7-4bb1-b716-c397d3329687-etc-kubernetes\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.269629 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269392 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-host-kubelet\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.269629 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.269513 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-run-ovn\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.270444 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.270412 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.270673 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.270480 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-ovn-node-metrics-cert\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.271889 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.271828 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/09c54c4c-17f5-47a7-8d09-9f185f561093-agent-certs\") pod \"konnectivity-agent-qgfwl\" (UID: \"09c54c4c-17f5-47a7-8d09-9f185f561093\") " pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.280526 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.280426 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tbp\" (UniqueName: \"kubernetes.io/projected/85112b71-3ff2-41c6-a9e5-73d2d24b11d2-kube-api-access-m7tbp\") pod \"iptables-alerter-x2hbm\" (UID: \"85112b71-3ff2-41c6-a9e5-73d2d24b11d2\") " pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.280668 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.280624 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlhl\" (UniqueName: \"kubernetes.io/projected/337f5c5a-79e7-4bb1-b716-c397d3329687-kube-api-access-bzlhl\") pod \"multus-w72xx\" (UID: \"337f5c5a-79e7-4bb1-b716-c397d3329687\") " pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.280734 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.280716 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99vrc\" (UniqueName: \"kubernetes.io/projected/05275d90-5a9a-4184-8ab2-1dd8cfb011e6-kube-api-access-99vrc\") pod \"ovnkube-node-99dbz\" (UID: \"05275d90-5a9a-4184-8ab2-1dd8cfb011e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.280816 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.280788 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pr7\" (UniqueName: \"kubernetes.io/projected/259030fd-b0aa-4c90-b2b6-b3a82c8e994c-kube-api-access-52pr7\") pod \"node-ca-lnvms\" (UID: \"259030fd-b0aa-4c90-b2b6-b3a82c8e994c\") " pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.281247 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.281221 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqcs\" (UniqueName: \"kubernetes.io/projected/b6d46487-c70a-4f24-95f0-3c7ccc5fa143-kube-api-access-hpqcs\") pod \"multus-additional-cni-plugins-jdtrs\" (UID: \"b6d46487-c70a-4f24-95f0-3c7ccc5fa143\") " pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.354440 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.354405 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" Sep 15 12:25:21.361531 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.361503 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-68hjm" Sep 15 12:25:21.368741 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.368711 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxrzq" Sep 15 12:25:21.369379 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.369340 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363be6bc_33bd_4256_92e9_d8c4b3ad4787.slice/crio-30a26d012a653993ed1eae3ee0ea7ed018e2b18aff395a20a97f174c63a095ea WatchSource:0}: Error finding container 30a26d012a653993ed1eae3ee0ea7ed018e2b18aff395a20a97f174c63a095ea: Status 404 returned error can't find the container with id 30a26d012a653993ed1eae3ee0ea7ed018e2b18aff395a20a97f174c63a095ea Sep 15 12:25:21.373418 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.373174 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1580dcd6_b90a_491e_a1dc_f4cae36584a6.slice/crio-7e911a9445fc5d937a34b493ea692831e7338e7c7add0eb205c0319b3b122c92 WatchSource:0}: Error finding container 7e911a9445fc5d937a34b493ea692831e7338e7c7add0eb205c0319b3b122c92: Status 404 returned error can't find the container with id 7e911a9445fc5d937a34b493ea692831e7338e7c7add0eb205c0319b3b122c92 Sep 15 12:25:21.374751 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.374726 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w72xx" Sep 15 12:25:21.380786 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.380751 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:21.381443 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.381415 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c623cb_ff74_44c7_bdd3_6e3b23bf5b55.slice/crio-1edd35a18a09e9e079f5157f04c7519b1dbbcb84a291a226d57cbae7d20dd402 WatchSource:0}: Error finding container 1edd35a18a09e9e079f5157f04c7519b1dbbcb84a291a226d57cbae7d20dd402: Status 404 returned error can't find the container with id 1edd35a18a09e9e079f5157f04c7519b1dbbcb84a291a226d57cbae7d20dd402 Sep 15 12:25:21.386750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.386561 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:21.393228 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.393198 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05275d90_5a9a_4184_8ab2_1dd8cfb011e6.slice/crio-fb1a073c65f9f4ed15002f69d2a24b78fb41817abe2fa0a9d44918f344b508e0 WatchSource:0}: Error finding container fb1a073c65f9f4ed15002f69d2a24b78fb41817abe2fa0a9d44918f344b508e0: Status 404 returned error can't find the container with id fb1a073c65f9f4ed15002f69d2a24b78fb41817abe2fa0a9d44918f344b508e0 Sep 15 12:25:21.393331 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.393272 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnvms" Sep 15 12:25:21.400465 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.400441 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x2hbm" Sep 15 12:25:21.400811 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.400785 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c54c4c_17f5_47a7_8d09_9f185f561093.slice/crio-ebd725012985c74471418043af2c01ca29f763644472d93d04e3b4083ffb22e3 WatchSource:0}: Error finding container ebd725012985c74471418043af2c01ca29f763644472d93d04e3b4083ffb22e3: Status 404 returned error can't find the container with id ebd725012985c74471418043af2c01ca29f763644472d93d04e3b4083ffb22e3 Sep 15 12:25:21.405298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.405147 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" Sep 15 12:25:21.405672 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.405646 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259030fd_b0aa_4c90_b2b6_b3a82c8e994c.slice/crio-cca9bbead131c0284edd62730f06f1fef3ba8e3edd0d45eeb112712dd97bdaec WatchSource:0}: Error finding container cca9bbead131c0284edd62730f06f1fef3ba8e3edd0d45eeb112712dd97bdaec: Status 404 returned error can't find the container with id cca9bbead131c0284edd62730f06f1fef3ba8e3edd0d45eeb112712dd97bdaec Sep 15 12:25:21.412965 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.412937 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85112b71_3ff2_41c6_a9e5_73d2d24b11d2.slice/crio-1899aa81d07085cae351531148d144c04bd39ba7096e155e14b82c49e44e714d WatchSource:0}: Error finding container 1899aa81d07085cae351531148d144c04bd39ba7096e155e14b82c49e44e714d: Status 404 returned error can't find the container with id 1899aa81d07085cae351531148d144c04bd39ba7096e155e14b82c49e44e714d Sep 15 12:25:21.417797 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:21.417773 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d46487_c70a_4f24_95f0_3c7ccc5fa143.slice/crio-73532a8098ae75794ddfea0f2187178dc66fd9349d81c62778d7b02eab62dc19 WatchSource:0}: Error finding container 73532a8098ae75794ddfea0f2187178dc66fd9349d81c62778d7b02eab62dc19: Status 404 returned error can't find the container with id 73532a8098ae75794ddfea0f2187178dc66fd9349d81c62778d7b02eab62dc19 Sep 15 12:25:21.511217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.511185 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 12:25:21.668873 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.668783 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:21.669029 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.668955 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:21.669071 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.669033 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs podName:f4a9b48e-794e-41c2-920e-2ff5156631da nodeName:}" failed. No retries permitted until 2026-09-15 12:25:22.669015797 +0000 UTC m=+4.023862100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs") pod "network-metrics-daemon-bw2fs" (UID: "f4a9b48e-794e-41c2-920e-2ff5156631da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:21.769682 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.769637 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:21.769885 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:21.769714 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:21.769885 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.769830 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:21.769885 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.769838 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 12:25:21.769885 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.769857 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 12:25:21.769885 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.769870 2550 projected.go:196] Error preparing data for projected volume kube-api-access-zf8g2 for pod openshift-network-diagnostics/network-check-target-dr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:21.770162 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.769899 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret podName:ea8d9432-e366-40c2-81fd-b478d05a1df7 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:22.769880479 +0000 UTC m=+4.124726804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret") pod "global-pull-secret-syncer-px8td" (UID: "ea8d9432-e366-40c2-81fd-b478d05a1df7") : object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:21.770162 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:21.769951 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2 podName:e1339667-307a-4bfd-a75c-a9d3e8d81e2d nodeName:}" failed. No retries permitted until 2026-09-15 12:25:22.769938352 +0000 UTC m=+4.124784656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf8g2" (UniqueName: "kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2") pod "network-check-target-dr79g" (UID: "e1339667-307a-4bfd-a75c-a9d3e8d81e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:22.091639 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.091601 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 12:20:21 +0000 UTC" deadline="2028-03-08 10:23:05.248292292 +0000 UTC" Sep 15 12:25:22.091639 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.091635 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12957h57m43.156661496s" Sep 15 12:25:22.147237 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.147197 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:22.147442 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.147353 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:22.156435 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.156386 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnvms" event={"ID":"259030fd-b0aa-4c90-b2b6-b3a82c8e994c","Type":"ContainerStarted","Data":"cca9bbead131c0284edd62730f06f1fef3ba8e3edd0d45eeb112712dd97bdaec"} Sep 15 12:25:22.157773 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.157741 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qgfwl" event={"ID":"09c54c4c-17f5-47a7-8d09-9f185f561093","Type":"ContainerStarted","Data":"ebd725012985c74471418043af2c01ca29f763644472d93d04e3b4083ffb22e3"} Sep 15 12:25:22.159497 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.159468 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x2hbm" event={"ID":"85112b71-3ff2-41c6-a9e5-73d2d24b11d2","Type":"ContainerStarted","Data":"1899aa81d07085cae351531148d144c04bd39ba7096e155e14b82c49e44e714d"} Sep 15 12:25:22.161236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.161189 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"fb1a073c65f9f4ed15002f69d2a24b78fb41817abe2fa0a9d44918f344b508e0"} Sep 15 12:25:22.163147 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.163120 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w72xx" event={"ID":"337f5c5a-79e7-4bb1-b716-c397d3329687","Type":"ContainerStarted","Data":"ba7a3719747468a8690271a33e1d539cf36c868fe9928623643b2986d5a5d707"} Sep 15 12:25:22.164931 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.164905 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxrzq" event={"ID":"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55","Type":"ContainerStarted","Data":"1edd35a18a09e9e079f5157f04c7519b1dbbcb84a291a226d57cbae7d20dd402"} Sep 15 12:25:22.167037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.167007 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-68hjm" event={"ID":"1580dcd6-b90a-491e-a1dc-f4cae36584a6","Type":"ContainerStarted","Data":"7e911a9445fc5d937a34b493ea692831e7338e7c7add0eb205c0319b3b122c92"} Sep 15 12:25:22.169058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.169034 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" event={"ID":"363be6bc-33bd-4256-92e9-d8c4b3ad4787","Type":"ContainerStarted","Data":"30a26d012a653993ed1eae3ee0ea7ed018e2b18aff395a20a97f174c63a095ea"} Sep 15 12:25:22.171900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.171684 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerStarted","Data":"73532a8098ae75794ddfea0f2187178dc66fd9349d81c62778d7b02eab62dc19"} Sep 15 12:25:22.676089 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.676053 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:22.676271 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.676248 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:22.676336 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.676324 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs podName:f4a9b48e-794e-41c2-920e-2ff5156631da nodeName:}" failed. No retries permitted until 2026-09-15 12:25:24.676303339 +0000 UTC m=+6.031149644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs") pod "network-metrics-daemon-bw2fs" (UID: "f4a9b48e-794e-41c2-920e-2ff5156631da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:22.776952 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.776908 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:22.777144 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:22.777005 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:22.777205 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.777162 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 12:25:22.777205 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.777183 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 12:25:22.777205 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.777198 2550 projected.go:196] Error preparing data for projected volume kube-api-access-zf8g2 for pod openshift-network-diagnostics/network-check-target-dr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:22.777407 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.777266 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2 podName:e1339667-307a-4bfd-a75c-a9d3e8d81e2d nodeName:}" failed. No retries permitted until 2026-09-15 12:25:24.777245934 +0000 UTC m=+6.132092260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf8g2" (UniqueName: "kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2") pod "network-check-target-dr79g" (UID: "e1339667-307a-4bfd-a75c-a9d3e8d81e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:22.778067 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.777739 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:22.778067 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:22.777816 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret podName:ea8d9432-e366-40c2-81fd-b478d05a1df7 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:24.777797728 +0000 UTC m=+6.132644035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret") pod "global-pull-secret-syncer-px8td" (UID: "ea8d9432-e366-40c2-81fd-b478d05a1df7") : object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:23.092823 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:23.092733 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 12:20:21 +0000 UTC" deadline="2028-03-29 04:47:30.737874784 +0000 UTC" Sep 15 12:25:23.092823 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:23.092778 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13456h22m7.645102078s" Sep 15 12:25:23.152189 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:23.148933 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:23.152189 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:23.149103 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:23.152189 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:23.149557 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:23.152189 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:23.149669 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:23.189963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:23.189883 2550 generic.go:356] "Generic (PLEG): container finished" podID="ab53c1e711a5972e218bac8960973b27" containerID="3b3febab2cef4e56b1cca8f831446f76d1c9ac0ac6e072962d7db64194588ae1" exitCode=0 Sep 15 12:25:23.189963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:23.189955 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"ab53c1e711a5972e218bac8960973b27","Type":"ContainerDied","Data":"3b3febab2cef4e56b1cca8f831446f76d1c9ac0ac6e072962d7db64194588ae1"} Sep 15 12:25:24.147963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:24.147609 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:24.148430 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.147993 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:24.216822 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:24.216088 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" event={"ID":"ab53c1e711a5972e218bac8960973b27","Type":"ContainerStarted","Data":"a65c2fb2bab8fe2555b7a4d7a7bbbb24c2eda5cce54fa47ee43c61303c523e95"} Sep 15 12:25:24.693530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:24.693424 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:24.693746 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.693661 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:24.693814 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.693753 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs podName:f4a9b48e-794e-41c2-920e-2ff5156631da nodeName:}" failed. No retries permitted until 2026-09-15 12:25:28.693733033 +0000 UTC m=+10.048579340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs") pod "network-metrics-daemon-bw2fs" (UID: "f4a9b48e-794e-41c2-920e-2ff5156631da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:24.794242 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:24.794339 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.794531 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.794551 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.794564 2550 projected.go:196] Error preparing data for projected volume kube-api-access-zf8g2 for pod openshift-network-diagnostics/network-check-target-dr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.794624 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2 podName:e1339667-307a-4bfd-a75c-a9d3e8d81e2d nodeName:}" failed. No retries permitted until 2026-09-15 12:25:28.794604356 +0000 UTC m=+10.149450675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf8g2" (UniqueName: "kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2") pod "network-check-target-dr79g" (UID: "e1339667-307a-4bfd-a75c-a9d3e8d81e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.795023 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:24.795116 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:24.795070 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret podName:ea8d9432-e366-40c2-81fd-b478d05a1df7 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:28.795055159 +0000 UTC m=+10.149901477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret") pod "global-pull-secret-syncer-px8td" (UID: "ea8d9432-e366-40c2-81fd-b478d05a1df7") : object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:25.148063 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:25.147980 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:25.148565 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:25.148130 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:25.148565 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:25.148187 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:25.148565 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:25.148325 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:26.147295 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:26.147261 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:26.147539 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:26.147421 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:27.158357 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:27.157350 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:27.158357 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:27.157840 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:27.158357 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:27.158259 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:27.158993 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:27.158448 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:28.148154 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:28.148059 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:28.148337 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.148218 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:28.728376 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:28.728276 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:28.729049 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.728484 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:28.729049 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.728568 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs podName:f4a9b48e-794e-41c2-920e-2ff5156631da nodeName:}" failed. No retries permitted until 2026-09-15 12:25:36.728544886 +0000 UTC m=+18.083391203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs") pod "network-metrics-daemon-bw2fs" (UID: "f4a9b48e-794e-41c2-920e-2ff5156631da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:28.829577 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:28.829649 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.829813 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.829883 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret podName:ea8d9432-e366-40c2-81fd-b478d05a1df7 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:36.829864045 +0000 UTC m=+18.184710364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret") pod "global-pull-secret-syncer-px8td" (UID: "ea8d9432-e366-40c2-81fd-b478d05a1df7") : object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.830102 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.830123 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.830135 2550 projected.go:196] Error preparing data for projected volume kube-api-access-zf8g2 for pod openshift-network-diagnostics/network-check-target-dr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:28.830232 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:28.830177 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2 podName:e1339667-307a-4bfd-a75c-a9d3e8d81e2d nodeName:}" failed. No retries permitted until 2026-09-15 12:25:36.830162417 +0000 UTC m=+18.185008729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf8g2" (UniqueName: "kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2") pod "network-check-target-dr79g" (UID: "e1339667-307a-4bfd-a75c-a9d3e8d81e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:29.151544 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:29.151466 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:29.151753 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:29.151617 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:29.151753 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:29.151656 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:29.151753 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:29.151743 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:30.148486 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:30.147926 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:30.148486 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:30.148080 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:31.151455 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:31.151420 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:31.151905 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:31.151476 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:31.151905 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:31.151583 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:31.151905 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:31.151696 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:32.147998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:32.147965 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:32.148170 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:32.148089 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:33.147902 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:33.147865 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:33.148356 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:33.147865 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:33.148356 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:33.148001 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:33.148356 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:33.148050 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:34.147273 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:34.147238 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:34.147472 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:34.147364 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:35.150698 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:35.150665 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:35.150698 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:35.150702 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:35.151160 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:35.150792 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:35.151160 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:35.150913 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:36.147748 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:36.147710 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:36.147950 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.147840 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:36.790973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:36.790934 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:36.791528 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.791115 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:36.791528 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.791199 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs podName:f4a9b48e-794e-41c2-920e-2ff5156631da nodeName:}" failed. No retries permitted until 2026-09-15 12:25:52.791178722 +0000 UTC m=+34.146025035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs") pod "network-metrics-daemon-bw2fs" (UID: "f4a9b48e-794e-41c2-920e-2ff5156631da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:36.891301 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:36.891258 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:36.891474 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:36.891325 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:36.891474 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.891418 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:36.891474 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.891440 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 12:25:36.891474 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.891455 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 12:25:36.891474 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.891466 2550 projected.go:196] Error preparing data for projected volume kube-api-access-zf8g2 for pod openshift-network-diagnostics/network-check-target-dr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:36.891639 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.891485 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret podName:ea8d9432-e366-40c2-81fd-b478d05a1df7 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:52.891470072 +0000 UTC m=+34.246316374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret") pod "global-pull-secret-syncer-px8td" (UID: "ea8d9432-e366-40c2-81fd-b478d05a1df7") : object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:36.891639 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:36.891506 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2 podName:e1339667-307a-4bfd-a75c-a9d3e8d81e2d nodeName:}" failed. No retries permitted until 2026-09-15 12:25:52.891495667 +0000 UTC m=+34.246341967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf8g2" (UniqueName: "kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2") pod "network-check-target-dr79g" (UID: "e1339667-307a-4bfd-a75c-a9d3e8d81e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:37.150494 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:37.150464 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:37.150653 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:37.150586 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:37.150653 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:37.150616 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:37.150745 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:37.150666 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:38.147735 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:38.147694 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:38.148199 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:38.147831 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:39.148446 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:39.148411 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:39.148906 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:39.148514 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:39.148906 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:39.148575 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:39.148906 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:39.148721 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:40.147649 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:40.147611 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:40.147840 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:40.147724 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:41.147871 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.147678 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:41.148734 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.147678 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:41.148734 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:41.147984 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:41.148734 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:41.148028 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:41.255126 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.255091 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxrzq" event={"ID":"d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55","Type":"ContainerStarted","Data":"926cbe270a0b689032fb19bd6d2c1220ca2fe9be6099f7e3bf74bba6f57414ba"} Sep 15 12:25:41.256520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.256494 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-68hjm" event={"ID":"1580dcd6-b90a-491e-a1dc-f4cae36584a6","Type":"ContainerStarted","Data":"639e4a9f28867e3dc93ab9a8cbbb8d8d7767de6f10459fc03febb790c33b7f15"} Sep 15 12:25:41.258134 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.258111 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" event={"ID":"363be6bc-33bd-4256-92e9-d8c4b3ad4787","Type":"ContainerStarted","Data":"d4f7327af2041d75bf33c425db4691a1d9f9d272ca4fd6717fc62cd4966ec954"} Sep 15 12:25:41.259551 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.259519 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" event={"ID":"f58eb2157143886588c630e14dfef629","Type":"ContainerStarted","Data":"adf44d39f6747c78473b979c8434c938abae097f40ad9bf89c2b9b12b1eb3bd5"} Sep 15 12:25:41.261159 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.261139 2550 generic.go:356] "Generic (PLEG): container finished" podID="b6d46487-c70a-4f24-95f0-3c7ccc5fa143" containerID="f31036ceca1d83cfc280c4f180706089bc97bbc8acdc9c00718c15353af2ff96" exitCode=0 Sep 15 12:25:41.261244 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.261224 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerDied","Data":"f31036ceca1d83cfc280c4f180706089bc97bbc8acdc9c00718c15353af2ff96"} Sep 15 12:25:41.263519 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.263041 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnvms" event={"ID":"259030fd-b0aa-4c90-b2b6-b3a82c8e994c","Type":"ContainerStarted","Data":"44788c75ee13a3c202c764812048d39e169d0c997dc596e1a8106db01fff130c"} Sep 15 12:25:41.264692 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.264667 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qgfwl" event={"ID":"09c54c4c-17f5-47a7-8d09-9f185f561093","Type":"ContainerStarted","Data":"864c8a6c74d501241ea4ae858c2e903ca464a4ed05a96590492c060805c5b0c4"} Sep 15 12:25:41.268260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.268239 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"1f713f6ef8d8bc0285fbff038a5a3cc887ef998e0fbb30cedde31bafc5e10cd7"} Sep 15 12:25:41.268349 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.268303 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"c3be68ead9113442cfe923a6ca089bb36c85556029b1568d0855091217ccf407"} Sep 15 12:25:41.268349 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.268335 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"1b5c190c0238e825f5d5412c95fcce2250650cc76e4a9819359b35280a89c9f6"} Sep 15 12:25:41.268469 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.268363 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"2651f7c2951ba795d4e1dba4a1856bf7baf535ff3640ed19399da2d04ba94c75"} Sep 15 12:25:41.268469 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.268421 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"639f4bceacb054500d577473949f3b8b3c319ac55758e610bca215d1db41570f"} Sep 15 12:25:41.270037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.270013 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w72xx" event={"ID":"337f5c5a-79e7-4bb1-b716-c397d3329687","Type":"ContainerStarted","Data":"12492ceb1f688021d8818ef17e6f34c47d54b4f20d8ee8d67ecc37b1ffa6a5b4"} Sep 15 12:25:41.270159 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.270116 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fxrzq" podStartSLOduration=1.989643957 podStartE2EDuration="21.270103421s" totalImagesPullingTime="19.280459464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:41.269901939 +0000 UTC m=+22.624748262" watchObservedRunningTime="2026-09-15 12:25:41.270103421 +0000 UTC m=+22.624949743" Sep 15 12:25:41.270546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.270507 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-242.ec2.internal" podStartSLOduration=20.270495971 podStartE2EDuration="20.270495971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:24.23431236 +0000 UTC m=+5.589158687" watchObservedRunningTime="2026-09-15 12:25:41.270495971 +0000 UTC m=+22.625342295" Sep 15 12:25:41.300737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.300686 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-242.ec2.internal" podStartSLOduration=20.300669275 podStartE2EDuration="20.300669275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:41.300264288 +0000 UTC m=+22.655110613" watchObservedRunningTime="2026-09-15 12:25:41.300669275 +0000 UTC m=+22.655515598" Sep 15 12:25:41.325492 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.325443 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qgfwl" podStartSLOduration=2.017201225 podStartE2EDuration="21.325425187s" totalImagesPullingTime="19.308223962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:41.313046444 +0000 UTC m=+22.667892768" watchObservedRunningTime="2026-09-15 12:25:41.325425187 +0000 UTC m=+22.680271509" Sep 15 12:25:41.325832 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.325807 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lnvms" podStartSLOduration=2.042421008 podStartE2EDuration="21.32580229s" totalImagesPullingTime="19.283381282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:41.325568018 +0000 UTC m=+22.680414334" watchObservedRunningTime="2026-09-15 12:25:41.32580229 +0000 UTC m=+22.680648604" Sep 15 12:25:41.341924 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.341870 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-68hjm" podStartSLOduration=2.000894918 podStartE2EDuration="21.341847362s" totalImagesPullingTime="19.340952444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:41.341273472 +0000 UTC m=+22.696119796" watchObservedRunningTime="2026-09-15 12:25:41.341847362 +0000 UTC m=+22.696693688" Sep 15 12:25:41.362967 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:41.360536 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w72xx" podStartSLOduration=1.976112399 podStartE2EDuration="21.360507002s" totalImagesPullingTime="19.384394603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:41.360064322 +0000 UTC m=+22.714910647" watchObservedRunningTime="2026-09-15 12:25:41.360507002 +0000 UTC m=+22.715353325" Sep 15 12:25:42.147664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:42.147619 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:42.147849 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:42.147734 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:42.274569 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:42.274518 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x2hbm" event={"ID":"85112b71-3ff2-41c6-a9e5-73d2d24b11d2","Type":"ContainerStarted","Data":"aefe13a31440351eebc37cb518d2fe7d1b4f05e5f6b7dbcbd823abf320c6a60b"} Sep 15 12:25:42.278294 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:42.278252 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"444b8a88d94c3892521dcfffe716f27c2175c2aa557ebfc772a2d89116e5201c"} Sep 15 12:25:42.289692 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:42.289646 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x2hbm" podStartSLOduration=3.013527634 podStartE2EDuration="22.289629066s" totalImagesPullingTime="19.276101432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:42.289555422 +0000 UTC m=+23.644401746" watchObservedRunningTime="2026-09-15 12:25:42.289629066 +0000 UTC m=+23.644475389" Sep 15 12:25:42.397361 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:42.397331 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 12:25:42.792574 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:42.792532 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:42.793234 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:42.793211 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:43.122641 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:43.122491 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T12:25:42.397355482Z","UUID":"5d511314-5ca0-4c8a-bae5-cfebff802a66","Handler":null,"Name":"","Endpoint":""} Sep 15 12:25:43.124265 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:43.124184 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 12:25:43.124265 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:43.124222 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 12:25:43.147467 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:43.147433 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:43.147651 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:43.147571 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:43.147715 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:43.147677 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:43.147800 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:43.147781 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:43.282992 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:43.282954 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" event={"ID":"363be6bc-33bd-4256-92e9-d8c4b3ad4787","Type":"ContainerStarted","Data":"8d8bb37b3df90d664a3e40cf1ac025c5dd5b780195389f0747f87b8bc458dd8a"} Sep 15 12:25:44.147758 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:44.147723 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:44.147927 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:44.147839 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:44.290451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:44.290411 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"dd47455c832d8f9f59ba43921253455aafde4192df67c50be8efdd79d6c02431"} Sep 15 12:25:44.292737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:44.292713 2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 12:25:44.292883 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:44.292713 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" event={"ID":"363be6bc-33bd-4256-92e9-d8c4b3ad4787","Type":"ContainerStarted","Data":"464a52f38c31b250a939f9c239ed03b9e0fe80c4946929641f23ce5739a71680"} Sep 15 12:25:44.310703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:44.310640 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t7lr9" podStartSLOduration=2.806120505 podStartE2EDuration="24.310623317s" totalImagesPullingTime="21.504502812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:44.310064691 +0000 UTC m=+25.664911014" watchObservedRunningTime="2026-09-15 12:25:44.310623317 +0000 UTC m=+25.665469642" Sep 15 12:25:45.147014 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:45.146961 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:45.147211 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:45.146972 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:45.147211 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:45.147092 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:45.147211 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:45.147181 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:46.148099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:46.147907 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:46.148716 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:46.148189 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:46.298968 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:46.298925 2550 generic.go:356] "Generic (PLEG): container finished" podID="b6d46487-c70a-4f24-95f0-3c7ccc5fa143" containerID="ea932b96a901ff2c3e0f7153e7a389f556c4a6d86375940b8bd92f880e31b5af" exitCode=0 Sep 15 12:25:46.299109 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:46.298984 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerDied","Data":"ea932b96a901ff2c3e0f7153e7a389f556c4a6d86375940b8bd92f880e31b5af"} Sep 15 12:25:46.303024 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:46.302999 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" event={"ID":"05275d90-5a9a-4184-8ab2-1dd8cfb011e6","Type":"ContainerStarted","Data":"bb87ecb35a406284121d8bda2e320349d36704e35942c8af875c850dbc9a0a83"} Sep 15 12:25:46.303303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:46.303288 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:46.318591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:46.318565 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:46.344714 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:46.342355 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" podStartSLOduration=7.002749368 podStartE2EDuration="26.342332228s" totalImagesPullingTime="19.33958286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:46.340632255 +0000 UTC m=+27.695478577" watchObservedRunningTime="2026-09-15 12:25:46.342332228 +0000 UTC m=+27.697178553" Sep 15 12:25:47.147845 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.147811 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:47.148008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.147811 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:47.148008 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:47.147942 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:47.148008 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:47.147986 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:47.305500 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.305472 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:47.305660 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.305566 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:47.320562 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.320540 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:25:47.732215 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.731935 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dr79g"] Sep 15 12:25:47.732374 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.732254 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:47.732374 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:47.732354 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:47.735243 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.735216 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bw2fs"] Sep 15 12:25:47.735366 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.735342 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:47.735544 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:47.735518 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:47.735993 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.735967 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-px8td"] Sep 15 12:25:47.736092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:47.736057 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:47.736162 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:47.736143 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:48.309750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:48.309717 2550 generic.go:356] "Generic (PLEG): container finished" podID="b6d46487-c70a-4f24-95f0-3c7ccc5fa143" containerID="33fac2e65da42459550652c7d847b554c4d513c0f16c9036075f2471e8141626" exitCode=0 Sep 15 12:25:48.310138 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:48.309806 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerDied","Data":"33fac2e65da42459550652c7d847b554c4d513c0f16c9036075f2471e8141626"} Sep 15 12:25:49.148020 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:49.147985 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:49.148207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:49.148100 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:49.148207 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:49.148116 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:49.148357 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:49.148215 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:49.148357 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:49.148254 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:49.148442 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:49.148369 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:50.316159 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:50.316127 2550 generic.go:356] "Generic (PLEG): container finished" podID="b6d46487-c70a-4f24-95f0-3c7ccc5fa143" containerID="b8a12d87af55fc067602f514cda19f0d408db83f8c41638a23a69a2de57ddade" exitCode=0 Sep 15 12:25:50.316159 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:50.316167 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerDied","Data":"b8a12d87af55fc067602f514cda19f0d408db83f8c41638a23a69a2de57ddade"} Sep 15 12:25:50.988976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:50.988942 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:50.989144 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:50.989093 2550 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 12:25:50.989650 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:50.989629 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qgfwl" Sep 15 12:25:51.147372 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:51.147342 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:51.147571 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:51.147351 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:51.147571 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:51.147444 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:51.147571 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:51.147465 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:51.147728 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:51.147571 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:51.147728 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:51.147642 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:52.817137 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:52.817101 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:52.817707 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.817276 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:52.817707 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.817363 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs podName:f4a9b48e-794e-41c2-920e-2ff5156631da nodeName:}" failed. No retries permitted until 2026-09-15 12:26:24.817340533 +0000 UTC m=+66.172186836 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs") pod "network-metrics-daemon-bw2fs" (UID: "f4a9b48e-794e-41c2-920e-2ff5156631da") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 12:25:52.917700 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:52.917662 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:52.917881 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:52.917726 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:52.917881 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.917827 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:52.917881 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.917839 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 12:25:52.917881 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.917856 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 12:25:52.917881 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.917869 2550 projected.go:196] Error preparing data for projected volume kube-api-access-zf8g2 for pod openshift-network-diagnostics/network-check-target-dr79g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:52.918087 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.917895 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret podName:ea8d9432-e366-40c2-81fd-b478d05a1df7 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:24.91787794 +0000 UTC m=+66.272724265 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret") pod "global-pull-secret-syncer-px8td" (UID: "ea8d9432-e366-40c2-81fd-b478d05a1df7") : object "kube-system"/"original-pull-secret" not registered Sep 15 12:25:52.918087 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:52.917913 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2 podName:e1339667-307a-4bfd-a75c-a9d3e8d81e2d nodeName:}" failed. No retries permitted until 2026-09-15 12:26:24.917904979 +0000 UTC m=+66.272751280 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zf8g2" (UniqueName: "kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2") pod "network-check-target-dr79g" (UID: "e1339667-307a-4bfd-a75c-a9d3e8d81e2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 12:25:53.150686 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.150601 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:53.150850 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.150700 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:53.150850 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.150725 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:53.150850 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.150715 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dr79g" podUID="e1339667-307a-4bfd-a75c-a9d3e8d81e2d" Sep 15 12:25:53.150850 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.150809 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-px8td" podUID="ea8d9432-e366-40c2-81fd-b478d05a1df7" Sep 15 12:25:53.151049 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.150878 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bw2fs" podUID="f4a9b48e-794e-41c2-920e-2ff5156631da" Sep 15 12:25:53.384642 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.384602 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-242.ec2.internal" event="NodeReady" Sep 15 12:25:53.384821 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.384737 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 12:25:53.420298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.420224 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bd8db4dc9-m8kbs"] Sep 15 12:25:53.424107 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.424081 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr"] Sep 15 12:25:53.424308 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.424284 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.426509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.426484 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 12:25:53.426600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.426523 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4xr84\"" Sep 15 12:25:53.426600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.426582 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 12:25:53.427108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.427089 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 12:25:53.427671 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.427653 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b"] Sep 15 12:25:53.427798 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.427783 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.430005 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.429985 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 15 12:25:53.430209 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.430194 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 15 12:25:53.431013 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.430987 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:25:53.431103 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.430990 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 15 12:25:53.431103 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.431038 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w4w2b\"" Sep 15 12:25:53.431103 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.431050 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb"] Sep 15 12:25:53.431345 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.431307 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:53.432984 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.432959 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 12:25:53.434108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.434052 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 15 12:25:53.434761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.434207 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 15 12:25:53.434761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.434362 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d2lz7\"" Sep 15 12:25:53.434886 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.434811 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8"] Sep 15 12:25:53.434980 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.434957 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:25:53.436427 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.435693 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" Sep 15 12:25:53.438607 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.438589 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4qwb9\"" Sep 15 12:25:53.439530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.439509 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:25:53.439634 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.439598 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 15 12:25:53.441316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.441298 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7chzc"] Sep 15 12:25:53.441461 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.441437 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.444396 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.444364 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 15 12:25:53.444396 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.444373 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 15 12:25:53.444840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.444418 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 15 12:25:53.444840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.444453 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr"] Sep 15 12:25:53.444840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.444626 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.447416 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.447397 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:25:53.447681 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.447445 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-448h6\"" Sep 15 12:25:53.448326 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.448301 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8"] Sep 15 12:25:53.449150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.449134 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bd8db4dc9-m8kbs"] Sep 15 12:25:53.450115 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.450097 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b"] Sep 15 12:25:53.451208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.451039 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7chzc"] Sep 15 12:25:53.452156 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.451916 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-m27xt"] Sep 15 12:25:53.452908 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.452887 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:25:53.453004 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.452958 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5nvvn\"" Sep 15 12:25:53.453004 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.452987 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 15 12:25:53.453130 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.453110 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 15 12:25:53.453240 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.453185 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 15 12:25:53.457491 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.457460 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" Sep 15 12:25:53.457491 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.457484 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 15 12:25:53.463044 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.463023 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-q7g4r\"" Sep 15 12:25:53.463675 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.463583 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 12:25:53.463947 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.463928 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 12:25:53.464021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.463960 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-m27xt"] Sep 15 12:25:53.468074 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.468027 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb"] Sep 15 12:25:53.483136 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.483110 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mtkjn"] Sep 15 12:25:53.487342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.487319 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ct9mg"] Sep 15 12:25:53.487479 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.487455 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.490679 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.490659 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:53.490679 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.490674 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 12:25:53.490861 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.490708 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 12:25:53.491156 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.491137 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pzg8w\"" Sep 15 12:25:53.493540 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.493518 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 12:25:53.493646 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.493561 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v4v6p\"" Sep 15 12:25:53.493883 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.493863 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 12:25:53.493972 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.493893 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 12:25:53.497393 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.497361 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mtkjn"] Sep 15 12:25:53.500193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.500172 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ct9mg"] Sep 15 12:25:53.521967 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.521931 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzh9\" (UniqueName: \"kubernetes.io/projected/f0996384-ef27-42a2-b215-daa5403e27b4-kube-api-access-rhzh9\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:53.522108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.521977 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-bound-sa-token\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522008 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c60e7f81-7308-463e-a85d-b46103f4fa9a-config\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.522108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522055 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9m2z\" (UniqueName: \"kubernetes.io/projected/93c9c4fb-9015-4d54-83c7-44e32e2fd8af-kube-api-access-l9m2z\") pod \"volume-data-source-validator-86695b7cc7-q5dcb\" (UID: \"93c9c4fb-9015-4d54-83c7-44e32e2fd8af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" Sep 15 12:25:53.522108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522090 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-image-registry-private-configuration\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-688xj\" (UniqueName: \"kubernetes.io/projected/c60e7f81-7308-463e-a85d-b46103f4fa9a-kube-api-access-688xj\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.522260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522170 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522198 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-certificates\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522219 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f41fba11-4f50-4a76-8daa-be90c630605c-serving-cert\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.522260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522250 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f41fba11-4f50-4a76-8daa-be90c630605c-config\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.522449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522272 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41fba11-4f50-4a76-8daa-be90c630605c-trusted-ca\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.522449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522286 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gpp8\" (UniqueName: \"kubernetes.io/projected/f41fba11-4f50-4a76-8daa-be90c630605c-kube-api-access-6gpp8\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.522449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522310 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hk9r\" (UniqueName: \"kubernetes.io/projected/d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514-kube-api-access-5hk9r\") pod \"network-check-source-6f8b768455-m27xt\" (UID: \"d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" Sep 15 12:25:53.522449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522333 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:53.522449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522348 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5zk\" (UniqueName: \"kubernetes.io/projected/18be3ccf-b37c-401c-a169-cc2820e6103f-kube-api-access-pp5zk\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.522449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522369 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-installation-pull-secrets\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522433 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-trusted-ca\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522452 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/18be3ccf-b37c-401c-a169-cc2820e6103f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.522702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522469 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80321730-1357-442e-9eb9-c72ce4cbf85b-ca-trust-extracted\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522492 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e7f81-7308-463e-a85d-b46103f4fa9a-serving-cert\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.522702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522524 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn4bv\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-kube-api-access-nn4bv\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.522702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.522552 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/18be3ccf-b37c-401c-a169-cc2820e6103f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.623174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623131 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-installation-pull-secrets\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.623369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623183 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b512fd0d-b99f-4f92-83f9-a7ace0c64f65-cert\") pod \"ingress-canary-ct9mg\" (UID: \"b512fd0d-b99f-4f92-83f9-a7ace0c64f65\") " pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:53.623369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623235 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-trusted-ca\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.623369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623261 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/18be3ccf-b37c-401c-a169-cc2820e6103f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.623369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623290 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80321730-1357-442e-9eb9-c72ce4cbf85b-ca-trust-extracted\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.623369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623315 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e7f81-7308-463e-a85d-b46103f4fa9a-serving-cert\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.623369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623336 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnzcm\" (UniqueName: \"kubernetes.io/projected/00f55c5a-30e5-465e-a207-67dd23c5523a-kube-api-access-fnzcm\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.623369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623365 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn4bv\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-kube-api-access-nn4bv\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.623727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623408 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/18be3ccf-b37c-401c-a169-cc2820e6103f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.623727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623447 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzh9\" (UniqueName: \"kubernetes.io/projected/f0996384-ef27-42a2-b215-daa5403e27b4-kube-api-access-rhzh9\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:53.623727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623471 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-bound-sa-token\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.623727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623500 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c60e7f81-7308-463e-a85d-b46103f4fa9a-config\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.623727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623529 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9m2z\" (UniqueName: \"kubernetes.io/projected/93c9c4fb-9015-4d54-83c7-44e32e2fd8af-kube-api-access-l9m2z\") pod \"volume-data-source-validator-86695b7cc7-q5dcb\" (UID: \"93c9c4fb-9015-4d54-83c7-44e32e2fd8af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" Sep 15 12:25:53.623727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623557 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-image-registry-private-configuration\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.624028 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.623785 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80321730-1357-442e-9eb9-c72ce4cbf85b-ca-trust-extracted\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.624534 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.624263 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-trusted-ca\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.624683 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hh7s\" (UniqueName: \"kubernetes.io/projected/b512fd0d-b99f-4f92-83f9-a7ace0c64f65-kube-api-access-5hh7s\") pod \"ingress-canary-ct9mg\" (UID: \"b512fd0d-b99f-4f92-83f9-a7ace0c64f65\") " pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.624869 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00f55c5a-30e5-465e-a207-67dd23c5523a-config-volume\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.624917 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-688xj\" (UniqueName: \"kubernetes.io/projected/c60e7f81-7308-463e-a85d-b46103f4fa9a-kube-api-access-688xj\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.624933 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c60e7f81-7308-463e-a85d-b46103f4fa9a-config\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.624734 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/18be3ccf-b37c-401c-a169-cc2820e6103f-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.624942 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f55c5a-30e5-465e-a207-67dd23c5523a-tmp-dir\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.625006 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.625030 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-certificates\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.625060 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f41fba11-4f50-4a76-8daa-be90c630605c-serving-cert\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.625096 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f41fba11-4f50-4a76-8daa-be90c630605c-config\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.625122 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41fba11-4f50-4a76-8daa-be90c630605c-trusted-ca\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.625143 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gpp8\" (UniqueName: \"kubernetes.io/projected/f41fba11-4f50-4a76-8daa-be90c630605c-kube-api-access-6gpp8\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.625303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.625166 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hk9r\" (UniqueName: \"kubernetes.io/projected/d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514-kube-api-access-5hk9r\") pod \"network-check-source-6f8b768455-m27xt\" (UID: \"d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" Sep 15 12:25:53.626017 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.625757 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 12:25:53.626017 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.625774 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bd8db4dc9-m8kbs: secret "image-registry-tls" not found Sep 15 12:25:53.626017 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.625840 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls podName:80321730-1357-442e-9eb9-c72ce4cbf85b nodeName:}" failed. No retries permitted until 2026-09-15 12:25:54.125822658 +0000 UTC m=+35.480668967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls") pod "image-registry-6bd8db4dc9-m8kbs" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b") : secret "image-registry-tls" not found Sep 15 12:25:53.626217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.626014 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f41fba11-4f50-4a76-8daa-be90c630605c-config\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.626217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.626142 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.626217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.626193 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:53.626410 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.626221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5zk\" (UniqueName: \"kubernetes.io/projected/18be3ccf-b37c-401c-a169-cc2820e6103f-kube-api-access-pp5zk\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.626410 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.626285 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f41fba11-4f50-4a76-8daa-be90c630605c-trusted-ca\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.626410 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.626405 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 12:25:53.626561 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.626473 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls podName:f0996384-ef27-42a2-b215-daa5403e27b4 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:54.126456643 +0000 UTC m=+35.481302959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5ct2b" (UID: "f0996384-ef27-42a2-b215-daa5403e27b4") : secret "samples-operator-tls" not found Sep 15 12:25:53.626561 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.626541 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-certificates\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.628030 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.627982 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c60e7f81-7308-463e-a85d-b46103f4fa9a-serving-cert\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.628129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.628098 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-installation-pull-secrets\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.628129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.628112 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-image-registry-private-configuration\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.628787 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.628741 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f41fba11-4f50-4a76-8daa-be90c630605c-serving-cert\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.628955 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.628933 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/18be3ccf-b37c-401c-a169-cc2820e6103f-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.632668 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.632611 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9m2z\" (UniqueName: \"kubernetes.io/projected/93c9c4fb-9015-4d54-83c7-44e32e2fd8af-kube-api-access-l9m2z\") pod \"volume-data-source-validator-86695b7cc7-q5dcb\" (UID: \"93c9c4fb-9015-4d54-83c7-44e32e2fd8af\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" Sep 15 12:25:53.632910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.632889 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn4bv\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-kube-api-access-nn4bv\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.633762 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.633194 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-bound-sa-token\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:53.633762 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.633710 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hk9r\" (UniqueName: \"kubernetes.io/projected/d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514-kube-api-access-5hk9r\") pod \"network-check-source-6f8b768455-m27xt\" (UID: \"d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" Sep 15 12:25:53.633912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.633770 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzh9\" (UniqueName: \"kubernetes.io/projected/f0996384-ef27-42a2-b215-daa5403e27b4-kube-api-access-rhzh9\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:53.634838 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.634812 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-688xj\" (UniqueName: \"kubernetes.io/projected/c60e7f81-7308-463e-a85d-b46103f4fa9a-kube-api-access-688xj\") pod \"service-ca-operator-cf7c776b7-84pvr\" (UID: \"c60e7f81-7308-463e-a85d-b46103f4fa9a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.635057 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.635038 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5zk\" (UniqueName: \"kubernetes.io/projected/18be3ccf-b37c-401c-a169-cc2820e6103f-kube-api-access-pp5zk\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n5dv8\" (UID: \"18be3ccf-b37c-401c-a169-cc2820e6103f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.635331 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.635312 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gpp8\" (UniqueName: \"kubernetes.io/projected/f41fba11-4f50-4a76-8daa-be90c630605c-kube-api-access-6gpp8\") pod \"console-operator-696884fb9c-7chzc\" (UID: \"f41fba11-4f50-4a76-8daa-be90c630605c\") " pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.727479 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727399 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hh7s\" (UniqueName: \"kubernetes.io/projected/b512fd0d-b99f-4f92-83f9-a7ace0c64f65-kube-api-access-5hh7s\") pod \"ingress-canary-ct9mg\" (UID: \"b512fd0d-b99f-4f92-83f9-a7ace0c64f65\") " pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:53.727479 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727438 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00f55c5a-30e5-465e-a207-67dd23c5523a-config-volume\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.727479 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727477 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f55c5a-30e5-465e-a207-67dd23c5523a-tmp-dir\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.727722 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727535 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.727722 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727574 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b512fd0d-b99f-4f92-83f9-a7ace0c64f65-cert\") pod \"ingress-canary-ct9mg\" (UID: \"b512fd0d-b99f-4f92-83f9-a7ace0c64f65\") " pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:53.727722 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727618 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnzcm\" (UniqueName: \"kubernetes.io/projected/00f55c5a-30e5-465e-a207-67dd23c5523a-kube-api-access-fnzcm\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.727722 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.727650 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 12:25:53.727918 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:53.727746 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls podName:00f55c5a-30e5-465e-a207-67dd23c5523a nodeName:}" failed. No retries permitted until 2026-09-15 12:25:54.227722945 +0000 UTC m=+35.582569258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls") pod "dns-default-mtkjn" (UID: "00f55c5a-30e5-465e-a207-67dd23c5523a") : secret "dns-default-metrics-tls" not found Sep 15 12:25:53.727918 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727894 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f55c5a-30e5-465e-a207-67dd23c5523a-tmp-dir\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.728028 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.727980 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00f55c5a-30e5-465e-a207-67dd23c5523a-config-volume\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.730117 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.730093 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b512fd0d-b99f-4f92-83f9-a7ace0c64f65-cert\") pod \"ingress-canary-ct9mg\" (UID: \"b512fd0d-b99f-4f92-83f9-a7ace0c64f65\") " pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:53.735943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.735907 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnzcm\" (UniqueName: \"kubernetes.io/projected/00f55c5a-30e5-465e-a207-67dd23c5523a-kube-api-access-fnzcm\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:53.736103 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.736083 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hh7s\" (UniqueName: \"kubernetes.io/projected/b512fd0d-b99f-4f92-83f9-a7ace0c64f65-kube-api-access-5hh7s\") pod \"ingress-canary-ct9mg\" (UID: \"b512fd0d-b99f-4f92-83f9-a7ace0c64f65\") " pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:53.744942 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.744916 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" Sep 15 12:25:53.765287 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.765254 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" Sep 15 12:25:53.773958 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.773925 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" Sep 15 12:25:53.781787 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.781757 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:25:53.788702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.788594 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" Sep 15 12:25:53.806205 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:53.805918 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ct9mg" Sep 15 12:25:54.051569 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.051452 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb"] Sep 15 12:25:54.052760 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.052607 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr"] Sep 15 12:25:54.089643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.089610 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8"] Sep 15 12:25:54.090667 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.090569 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-7chzc"] Sep 15 12:25:54.119451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.119419 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ct9mg"] Sep 15 12:25:54.122365 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.122340 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-m27xt"] Sep 15 12:25:54.138215 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.138161 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:54.138349 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.138254 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:54.138349 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:54.138304 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 12:25:54.138349 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:54.138323 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bd8db4dc9-m8kbs: secret "image-registry-tls" not found Sep 15 12:25:54.138496 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:54.138402 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls podName:80321730-1357-442e-9eb9-c72ce4cbf85b nodeName:}" failed. No retries permitted until 2026-09-15 12:25:55.138370049 +0000 UTC m=+36.493216368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls") pod "image-registry-6bd8db4dc9-m8kbs" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b") : secret "image-registry-tls" not found Sep 15 12:25:54.138496 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:54.138404 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 12:25:54.138496 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:54.138458 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls podName:f0996384-ef27-42a2-b215-daa5403e27b4 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:55.138442629 +0000 UTC m=+36.493288931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5ct2b" (UID: "f0996384-ef27-42a2-b215-daa5403e27b4") : secret "samples-operator-tls" not found Sep 15 12:25:54.238953 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:54.238921 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:54.239137 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:54.239091 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 12:25:54.239199 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:54.239159 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls podName:00f55c5a-30e5-465e-a207-67dd23c5523a nodeName:}" failed. No retries permitted until 2026-09-15 12:25:55.239143894 +0000 UTC m=+36.593990194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls") pod "dns-default-mtkjn" (UID: "00f55c5a-30e5-465e-a207-67dd23c5523a") : secret "dns-default-metrics-tls" not found Sep 15 12:25:55.146146 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.146103 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:55.146736 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.146203 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:55.146736 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:55.146290 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 12:25:55.146736 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:55.146312 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bd8db4dc9-m8kbs: secret "image-registry-tls" not found Sep 15 12:25:55.146736 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:55.146367 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 12:25:55.146736 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:55.146394 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls podName:80321730-1357-442e-9eb9-c72ce4cbf85b nodeName:}" failed. No retries permitted until 2026-09-15 12:25:57.146359476 +0000 UTC m=+38.501205803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls") pod "image-registry-6bd8db4dc9-m8kbs" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b") : secret "image-registry-tls" not found Sep 15 12:25:55.146736 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:55.146572 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls podName:f0996384-ef27-42a2-b215-daa5403e27b4 nodeName:}" failed. No retries permitted until 2026-09-15 12:25:57.146422196 +0000 UTC m=+38.501268499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5ct2b" (UID: "f0996384-ef27-42a2-b215-daa5403e27b4") : secret "samples-operator-tls" not found Sep 15 12:25:55.148057 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.148035 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:25:55.148157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.148064 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:25:55.148157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.148035 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:25:55.150721 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.150700 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 12:25:55.150837 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.150808 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4plp\"" Sep 15 12:25:55.151731 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.151505 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 12:25:55.161849 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.161824 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mnpg9\"" Sep 15 12:25:55.246858 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:55.246822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:55.247046 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:55.246980 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 12:25:55.247109 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:55.247063 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls podName:00f55c5a-30e5-465e-a207-67dd23c5523a nodeName:}" failed. No retries permitted until 2026-09-15 12:25:57.247039505 +0000 UTC m=+38.601885827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls") pod "dns-default-mtkjn" (UID: "00f55c5a-30e5-465e-a207-67dd23c5523a") : secret "dns-default-metrics-tls" not found Sep 15 12:25:55.413531 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:55.413447 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41fba11_4f50_4a76_8daa_be90c630605c.slice/crio-b1a3268dc0a9535d6b3166b9a20538bd2561084e5b1e4078f379f69ace4647e5 WatchSource:0}: Error finding container b1a3268dc0a9535d6b3166b9a20538bd2561084e5b1e4078f379f69ace4647e5: Status 404 returned error can't find the container with id b1a3268dc0a9535d6b3166b9a20538bd2561084e5b1e4078f379f69ace4647e5 Sep 15 12:25:55.414689 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:55.414656 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd356f1bc_68d8_4fc8_bd1d_9bef3c7f4514.slice/crio-aba48df5fd54d7bae8a5b21c711adfd1e2c3cb132904ce57fccc829cb8596438 WatchSource:0}: Error finding container aba48df5fd54d7bae8a5b21c711adfd1e2c3cb132904ce57fccc829cb8596438: Status 404 returned error can't find the container with id aba48df5fd54d7bae8a5b21c711adfd1e2c3cb132904ce57fccc829cb8596438 Sep 15 12:25:55.417308 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:55.416594 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb512fd0d_b99f_4f92_83f9_a7ace0c64f65.slice/crio-a7a44859f623430b39164b1ddd081e05b757f6d90db225d5c68f5825ff37889a WatchSource:0}: Error finding container a7a44859f623430b39164b1ddd081e05b757f6d90db225d5c68f5825ff37889a: Status 404 returned error can't find the container with id a7a44859f623430b39164b1ddd081e05b757f6d90db225d5c68f5825ff37889a Sep 15 12:25:55.417308 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:55.417252 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18be3ccf_b37c_401c_a169_cc2820e6103f.slice/crio-b0d91fe532a1cdc15c3fc9c97958d546da01970ed00c05e369469d8b044434fd WatchSource:0}: Error finding container b0d91fe532a1cdc15c3fc9c97958d546da01970ed00c05e369469d8b044434fd: Status 404 returned error can't find the container with id b0d91fe532a1cdc15c3fc9c97958d546da01970ed00c05e369469d8b044434fd Sep 15 12:25:55.418098 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:55.417928 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60e7f81_7308_463e_a85d_b46103f4fa9a.slice/crio-d8ff879148f43274b20a51a33365a1cc2fdea10cab3cefe03abcbc81154bd715 WatchSource:0}: Error finding container d8ff879148f43274b20a51a33365a1cc2fdea10cab3cefe03abcbc81154bd715: Status 404 returned error can't find the container with id d8ff879148f43274b20a51a33365a1cc2fdea10cab3cefe03abcbc81154bd715 Sep 15 12:25:55.418270 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:25:55.418179 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c9c4fb_9015_4d54_83c7_44e32e2fd8af.slice/crio-cc430b60f58e6d24b5aa62f080b7dc33236e8ec9ba6718c695ead3d4f24cfae8 WatchSource:0}: Error finding container cc430b60f58e6d24b5aa62f080b7dc33236e8ec9ba6718c695ead3d4f24cfae8: Status 404 returned error can't find the container with id cc430b60f58e6d24b5aa62f080b7dc33236e8ec9ba6718c695ead3d4f24cfae8 Sep 15 12:25:56.334055 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.333707 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" event={"ID":"f41fba11-4f50-4a76-8daa-be90c630605c","Type":"ContainerStarted","Data":"b1a3268dc0a9535d6b3166b9a20538bd2561084e5b1e4078f379f69ace4647e5"} Sep 15 12:25:56.336537 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.336471 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" event={"ID":"c60e7f81-7308-463e-a85d-b46103f4fa9a","Type":"ContainerStarted","Data":"d8ff879148f43274b20a51a33365a1cc2fdea10cab3cefe03abcbc81154bd715"} Sep 15 12:25:56.339258 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.339193 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ct9mg" event={"ID":"b512fd0d-b99f-4f92-83f9-a7ace0c64f65","Type":"ContainerStarted","Data":"a7a44859f623430b39164b1ddd081e05b757f6d90db225d5c68f5825ff37889a"} Sep 15 12:25:56.341264 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.341228 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" event={"ID":"93c9c4fb-9015-4d54-83c7-44e32e2fd8af","Type":"ContainerStarted","Data":"cc430b60f58e6d24b5aa62f080b7dc33236e8ec9ba6718c695ead3d4f24cfae8"} Sep 15 12:25:56.343860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.343799 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" event={"ID":"18be3ccf-b37c-401c-a169-cc2820e6103f","Type":"ContainerStarted","Data":"b0d91fe532a1cdc15c3fc9c97958d546da01970ed00c05e369469d8b044434fd"} Sep 15 12:25:56.351073 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.350081 2550 generic.go:356] "Generic (PLEG): container finished" podID="b6d46487-c70a-4f24-95f0-3c7ccc5fa143" containerID="ffcd760bf887a698202cc2e35a7c3e1dcd14d9ca4036cfc043e689dd0d141edb" exitCode=0 Sep 15 12:25:56.351073 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.350165 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerDied","Data":"ffcd760bf887a698202cc2e35a7c3e1dcd14d9ca4036cfc043e689dd0d141edb"} Sep 15 12:25:56.355898 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:56.355541 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" event={"ID":"d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514","Type":"ContainerStarted","Data":"aba48df5fd54d7bae8a5b21c711adfd1e2c3cb132904ce57fccc829cb8596438"} Sep 15 12:25:57.163401 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:57.163341 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:25:57.163590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:57.163498 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:25:57.163707 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:57.163679 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 12:25:57.163707 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:57.163697 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bd8db4dc9-m8kbs: secret "image-registry-tls" not found Sep 15 12:25:57.163815 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:57.163755 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls podName:80321730-1357-442e-9eb9-c72ce4cbf85b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:01.163735382 +0000 UTC m=+42.518581697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls") pod "image-registry-6bd8db4dc9-m8kbs" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b") : secret "image-registry-tls" not found Sep 15 12:25:57.164183 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:57.164164 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 12:25:57.164263 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:57.164224 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls podName:f0996384-ef27-42a2-b215-daa5403e27b4 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:01.164206885 +0000 UTC m=+42.519053187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5ct2b" (UID: "f0996384-ef27-42a2-b215-daa5403e27b4") : secret "samples-operator-tls" not found Sep 15 12:25:57.264546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:57.264497 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:25:57.264748 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:57.264685 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 12:25:57.264835 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:57.264750 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls podName:00f55c5a-30e5-465e-a207-67dd23c5523a nodeName:}" failed. No retries permitted until 2026-09-15 12:26:01.264731617 +0000 UTC m=+42.619577947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls") pod "dns-default-mtkjn" (UID: "00f55c5a-30e5-465e-a207-67dd23c5523a") : secret "dns-default-metrics-tls" not found Sep 15 12:25:57.366273 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:57.366140 2550 generic.go:356] "Generic (PLEG): container finished" podID="b6d46487-c70a-4f24-95f0-3c7ccc5fa143" containerID="8234e07c033eeaf27e58fa792971b3af4ed1c8383f64c366425f3916921f5bf3" exitCode=0 Sep 15 12:25:57.366273 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:57.366217 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerDied","Data":"8234e07c033eeaf27e58fa792971b3af4ed1c8383f64c366425f3916921f5bf3"} Sep 15 12:25:58.373188 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:58.373152 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" event={"ID":"b6d46487-c70a-4f24-95f0-3c7ccc5fa143","Type":"ContainerStarted","Data":"e5f9464e17ec62a708b1ff4d858b140f91b2c4e2f90f6ea391cede90e6f61364"} Sep 15 12:25:58.400813 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:58.400762 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jdtrs" podStartSLOduration=7.185324262 podStartE2EDuration="38.400747142s" totalImagesPullingTime="31.21542288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:25:58.399858209 +0000 UTC m=+39.754704551" watchObservedRunningTime="2026-09-15 12:25:58.400747142 +0000 UTC m=+39.755593466" Sep 15 12:25:59.450523 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.450486 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g"] Sep 15 12:25:59.480763 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.480718 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g"] Sep 15 12:25:59.480929 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.480840 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:25:59.483270 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.483136 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 15 12:25:59.483270 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.483150 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 15 12:25:59.483270 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.483165 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-928zd\"" Sep 15 12:25:59.585825 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.585785 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/373725c3-1275-4adf-99dc-6a3cf09e620b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:25:59.586004 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.585887 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:25:59.687119 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.687067 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/373725c3-1275-4adf-99dc-6a3cf09e620b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:25:59.687305 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.687256 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:25:59.687367 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:59.687354 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 12:25:59.687440 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:25:59.687420 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert podName:373725c3-1275-4adf-99dc-6a3cf09e620b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:00.187406559 +0000 UTC m=+41.542252860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jv74g" (UID: "373725c3-1275-4adf-99dc-6a3cf09e620b") : secret "networking-console-plugin-cert" not found Sep 15 12:25:59.687968 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:25:59.687942 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/373725c3-1275-4adf-99dc-6a3cf09e620b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:00.191986 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:00.191952 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:00.192192 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:00.192082 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 12:26:00.192192 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:00.192142 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert podName:373725c3-1275-4adf-99dc-6a3cf09e620b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:01.19212916 +0000 UTC m=+42.546975460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jv74g" (UID: "373725c3-1275-4adf-99dc-6a3cf09e620b") : secret "networking-console-plugin-cert" not found Sep 15 12:26:01.200803 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:01.200618 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:26:01.200803 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:01.200667 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:26:01.200803 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:01.200697 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:01.201307 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.200875 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 12:26:01.201307 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.200924 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 12:26:01.201307 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.200880 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 12:26:01.201307 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.200973 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls podName:f0996384-ef27-42a2-b215-daa5403e27b4 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:09.200948608 +0000 UTC m=+50.555794927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5ct2b" (UID: "f0996384-ef27-42a2-b215-daa5403e27b4") : secret "samples-operator-tls" not found Sep 15 12:26:01.201307 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.200994 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert podName:373725c3-1275-4adf-99dc-6a3cf09e620b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:03.200985143 +0000 UTC m=+44.555831447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jv74g" (UID: "373725c3-1275-4adf-99dc-6a3cf09e620b") : secret "networking-console-plugin-cert" not found Sep 15 12:26:01.201307 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.200973 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bd8db4dc9-m8kbs: secret "image-registry-tls" not found Sep 15 12:26:01.201307 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.201029 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls podName:80321730-1357-442e-9eb9-c72ce4cbf85b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:09.201019721 +0000 UTC m=+50.555866039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls") pod "image-registry-6bd8db4dc9-m8kbs" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b") : secret "image-registry-tls" not found Sep 15 12:26:01.301371 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:01.301329 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:26:01.301600 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.301581 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 12:26:01.301678 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:01.301650 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls podName:00f55c5a-30e5-465e-a207-67dd23c5523a nodeName:}" failed. No retries permitted until 2026-09-15 12:26:09.301632646 +0000 UTC m=+50.656478947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls") pod "dns-default-mtkjn" (UID: "00f55c5a-30e5-465e-a207-67dd23c5523a") : secret "dns-default-metrics-tls" not found Sep 15 12:26:03.217700 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.217663 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:03.218171 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:03.217876 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 12:26:03.218171 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:03.217968 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert podName:373725c3-1275-4adf-99dc-6a3cf09e620b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:07.217932396 +0000 UTC m=+48.572778718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jv74g" (UID: "373725c3-1275-4adf-99dc-6a3cf09e620b") : secret "networking-console-plugin-cert" not found Sep 15 12:26:03.390794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.390755 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" event={"ID":"18be3ccf-b37c-401c-a169-cc2820e6103f","Type":"ContainerStarted","Data":"293592c90c23192d8011e250fdbe3d2d496b3a29557c00ff936c2edf17a89cdf"} Sep 15 12:26:03.392436 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.392402 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" event={"ID":"d356f1bc-68d8-4fc8-bd1d-9bef3c7f4514","Type":"ContainerStarted","Data":"0773daaae12664e6ca81123c02530e125068eeefd69f790dd95cdbc80697b6c4"} Sep 15 12:26:03.394174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.394151 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/0.log" Sep 15 12:26:03.394286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.394200 2550 generic.go:356] "Generic (PLEG): container finished" podID="f41fba11-4f50-4a76-8daa-be90c630605c" containerID="0249c5c2c7382032c7dbe1bd49dd1966519bf015db3ecc4ee5bac37975f8a2c5" exitCode=255 Sep 15 12:26:03.394286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.394236 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" event={"ID":"f41fba11-4f50-4a76-8daa-be90c630605c","Type":"ContainerDied","Data":"0249c5c2c7382032c7dbe1bd49dd1966519bf015db3ecc4ee5bac37975f8a2c5"} Sep 15 12:26:03.394491 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.394475 2550 scope.go:117] "RemoveContainer" containerID="0249c5c2c7382032c7dbe1bd49dd1966519bf015db3ecc4ee5bac37975f8a2c5" Sep 15 12:26:03.395868 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.395846 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" event={"ID":"c60e7f81-7308-463e-a85d-b46103f4fa9a","Type":"ContainerStarted","Data":"039f9a5442c8aec9f5913145fa54c0a96b9b121b415989aaddb768fa797b4ccd"} Sep 15 12:26:03.397909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.397891 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ct9mg" event={"ID":"b512fd0d-b99f-4f92-83f9-a7ace0c64f65","Type":"ContainerStarted","Data":"d488fbb07dd672e4f1b002d3656fce555d00d55e5090964fa516151c33057a00"} Sep 15 12:26:03.399495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.399475 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" event={"ID":"93c9c4fb-9015-4d54-83c7-44e32e2fd8af","Type":"ContainerStarted","Data":"96f8e55af5c07cca7f0ca3222253c01ebe3a0f20c4642c49ad464021304382cf"} Sep 15 12:26:03.407222 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.407175 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" podStartSLOduration=32.36200451 podStartE2EDuration="39.407157596s" totalImagesPullingTime="7.045153086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:03.406013567 +0000 UTC m=+44.760859905" watchObservedRunningTime="2026-09-15 12:26:03.407157596 +0000 UTC m=+44.762003920" Sep 15 12:26:03.437284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.437228 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" podStartSLOduration=32.392486121 podStartE2EDuration="39.437174121s" totalImagesPullingTime="7.044688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:03.420084924 +0000 UTC m=+44.774931249" watchObservedRunningTime="2026-09-15 12:26:03.437174121 +0000 UTC m=+44.792020442" Sep 15 12:26:03.437453 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.437341 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-q5dcb" podStartSLOduration=32.6903215 podStartE2EDuration="39.437334464s" totalImagesPullingTime="6.747012964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:03.437080176 +0000 UTC m=+44.791926500" watchObservedRunningTime="2026-09-15 12:26:03.437334464 +0000 UTC m=+44.792180788" Sep 15 12:26:03.458766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.458707 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-m27xt" podStartSLOduration=23.40277132 podStartE2EDuration="30.458685497s" totalImagesPullingTime="7.055914177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:03.456740124 +0000 UTC m=+44.811586447" watchObservedRunningTime="2026-09-15 12:26:03.458685497 +0000 UTC m=+44.813531821" Sep 15 12:26:03.523549 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.522595 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ct9mg" podStartSLOduration=3.476607286 podStartE2EDuration="10.522573458s" totalImagesPullingTime="7.045966172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:03.488753592 +0000 UTC m=+44.843599916" watchObservedRunningTime="2026-09-15 12:26:03.522573458 +0000 UTC m=+44.877419796" Sep 15 12:26:03.782659 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.782619 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:26:03.782814 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:03.782723 2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:26:04.404208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:04.404179 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:26:04.404662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:04.404646 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/0.log" Sep 15 12:26:04.404727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:04.404681 2550 generic.go:356] "Generic (PLEG): container finished" podID="f41fba11-4f50-4a76-8daa-be90c630605c" containerID="353c44e85dde98c48dcf240a320c509ba914ab1bccdc5e93d574c32a1cdbf047" exitCode=255 Sep 15 12:26:04.404727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:04.404714 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" event={"ID":"f41fba11-4f50-4a76-8daa-be90c630605c","Type":"ContainerDied","Data":"353c44e85dde98c48dcf240a320c509ba914ab1bccdc5e93d574c32a1cdbf047"} Sep 15 12:26:04.404815 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:04.404773 2550 scope.go:117] "RemoveContainer" containerID="0249c5c2c7382032c7dbe1bd49dd1966519bf015db3ecc4ee5bac37975f8a2c5" Sep 15 12:26:04.405047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:04.405010 2550 scope.go:117] "RemoveContainer" containerID="353c44e85dde98c48dcf240a320c509ba914ab1bccdc5e93d574c32a1cdbf047" Sep 15 12:26:04.405244 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:04.405201 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7chzc_openshift-console-operator(f41fba11-4f50-4a76-8daa-be90c630605c)\"" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" podUID="f41fba11-4f50-4a76-8daa-be90c630605c" Sep 15 12:26:05.412603 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:05.412576 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:26:05.412989 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:05.412902 2550 scope.go:117] "RemoveContainer" containerID="353c44e85dde98c48dcf240a320c509ba914ab1bccdc5e93d574c32a1cdbf047" Sep 15 12:26:05.413099 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:05.413080 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-7chzc_openshift-console-operator(f41fba11-4f50-4a76-8daa-be90c630605c)\"" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" podUID="f41fba11-4f50-4a76-8daa-be90c630605c" Sep 15 12:26:07.255266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:07.255224 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:07.255683 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:07.255347 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 12:26:07.255683 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:07.255420 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert podName:373725c3-1275-4adf-99dc-6a3cf09e620b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:15.255406125 +0000 UTC m=+56.610252426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jv74g" (UID: "373725c3-1275-4adf-99dc-6a3cf09e620b") : secret "networking-console-plugin-cert" not found Sep 15 12:26:09.272807 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:09.272773 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:26:09.273396 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:09.272812 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:26:09.273396 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:09.272916 2550 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 12:26:09.273396 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:09.272938 2550 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bd8db4dc9-m8kbs: secret "image-registry-tls" not found Sep 15 12:26:09.273396 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:09.272971 2550 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 12:26:09.273396 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:09.273008 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls podName:80321730-1357-442e-9eb9-c72ce4cbf85b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:25.27299207 +0000 UTC m=+66.627838373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls") pod "image-registry-6bd8db4dc9-m8kbs" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b") : secret "image-registry-tls" not found Sep 15 12:26:09.273396 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:09.273022 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls podName:f0996384-ef27-42a2-b215-daa5403e27b4 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:25.273016407 +0000 UTC m=+66.627862708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-5ct2b" (UID: "f0996384-ef27-42a2-b215-daa5403e27b4") : secret "samples-operator-tls" not found Sep 15 12:26:09.373669 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:09.373633 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:26:09.373823 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:09.373748 2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 12:26:09.373862 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:09.373837 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls podName:00f55c5a-30e5-465e-a207-67dd23c5523a nodeName:}" failed. No retries permitted until 2026-09-15 12:26:25.373822582 +0000 UTC m=+66.728668882 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls") pod "dns-default-mtkjn" (UID: "00f55c5a-30e5-465e-a207-67dd23c5523a") : secret "dns-default-metrics-tls" not found Sep 15 12:26:13.782047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:13.782013 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:26:13.782526 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:13.782063 2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:26:13.782526 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:13.782405 2550 scope.go:117] "RemoveContainer" containerID="353c44e85dde98c48dcf240a320c509ba914ab1bccdc5e93d574c32a1cdbf047" Sep 15 12:26:14.441941 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:14.441907 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:26:14.442131 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:14.441989 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" event={"ID":"f41fba11-4f50-4a76-8daa-be90c630605c","Type":"ContainerStarted","Data":"41d1accf4faec27b79e5b5cceeafc55b94f5510b5cd675cc55324fc821e010b8"} Sep 15 12:26:14.442284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:14.442254 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:26:14.446826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:14.446803 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" Sep 15 12:26:14.459148 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:14.459107 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-7chzc" podStartSLOduration=43.414998289 podStartE2EDuration="50.459095581s" totalImagesPullingTime="7.044097292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:14.458909772 +0000 UTC m=+55.813756096" watchObservedRunningTime="2026-09-15 12:26:14.459095581 +0000 UTC m=+55.813941903" Sep 15 12:26:15.324401 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:15.324351 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:15.324782 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:15.324491 2550 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 12:26:15.324782 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:15.324574 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert podName:373725c3-1275-4adf-99dc-6a3cf09e620b nodeName:}" failed. No retries permitted until 2026-09-15 12:26:31.32455807 +0000 UTC m=+72.679404384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-jv74g" (UID: "373725c3-1275-4adf-99dc-6a3cf09e620b") : secret "networking-console-plugin-cert" not found Sep 15 12:26:19.321950 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:19.321915 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99dbz" Sep 15 12:26:24.903335 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:24.903283 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:26:24.905751 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:24.905720 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 12:26:24.916859 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:24.916831 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4a9b48e-794e-41c2-920e-2ff5156631da-metrics-certs\") pod \"network-metrics-daemon-bw2fs\" (UID: \"f4a9b48e-794e-41c2-920e-2ff5156631da\") " pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:26:25.003916 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.003878 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:26:25.004106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.003927 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:26:25.006432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.006404 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 12:26:25.006579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.006561 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8g2\" (UniqueName: \"kubernetes.io/projected/e1339667-307a-4bfd-a75c-a9d3e8d81e2d-kube-api-access-zf8g2\") pod \"network-check-target-dr79g\" (UID: \"e1339667-307a-4bfd-a75c-a9d3e8d81e2d\") " pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:26:25.016836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.016803 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea8d9432-e366-40c2-81fd-b478d05a1df7-original-pull-secret\") pod \"global-pull-secret-syncer-px8td\" (UID: \"ea8d9432-e366-40c2-81fd-b478d05a1df7\") " pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:26:25.161174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.161084 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-px8td" Sep 15 12:26:25.170700 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.170678 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mnpg9\"" Sep 15 12:26:25.176009 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.175980 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4plp\"" Sep 15 12:26:25.178662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.178625 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:26:25.184597 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.184570 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bw2fs" Sep 15 12:26:25.305972 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.305939 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:26:25.306143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.305980 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:26:25.308267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.308242 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0996384-ef27-42a2-b215-daa5403e27b4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-5ct2b\" (UID: \"f0996384-ef27-42a2-b215-daa5403e27b4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:26:25.308364 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.308317 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"image-registry-6bd8db4dc9-m8kbs\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:26:25.406830 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.406794 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:26:25.408986 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.408955 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00f55c5a-30e5-465e-a207-67dd23c5523a-metrics-tls\") pod \"dns-default-mtkjn\" (UID: \"00f55c5a-30e5-465e-a207-67dd23c5523a\") " pod="openshift-dns/dns-default-mtkjn" Sep 15 12:26:25.438363 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.438293 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-px8td"] Sep 15 12:26:25.441286 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:25.441263 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8d9432_e366_40c2_81fd_b478d05a1df7.slice/crio-2d290c1a4b65b774103fa95e296e109ab2bc14e7b8dae28aacda8b65ceb7e178 WatchSource:0}: Error finding container 2d290c1a4b65b774103fa95e296e109ab2bc14e7b8dae28aacda8b65ceb7e178: Status 404 returned error can't find the container with id 2d290c1a4b65b774103fa95e296e109ab2bc14e7b8dae28aacda8b65ceb7e178 Sep 15 12:26:25.476303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.476272 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-px8td" event={"ID":"ea8d9432-e366-40c2-81fd-b478d05a1df7","Type":"ContainerStarted","Data":"2d290c1a4b65b774103fa95e296e109ab2bc14e7b8dae28aacda8b65ceb7e178"} Sep 15 12:26:25.485309 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.485279 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dr79g"] Sep 15 12:26:25.488412 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.488370 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bw2fs"] Sep 15 12:26:25.488770 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:25.488746 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1339667_307a_4bfd_a75c_a9d3e8d81e2d.slice/crio-30f093bbca81adf4d73a3c52663eaa2e454479a129a7acb308ab7c850ab8de98 WatchSource:0}: Error finding container 30f093bbca81adf4d73a3c52663eaa2e454479a129a7acb308ab7c850ab8de98: Status 404 returned error can't find the container with id 30f093bbca81adf4d73a3c52663eaa2e454479a129a7acb308ab7c850ab8de98 Sep 15 12:26:25.491133 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:25.491109 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a9b48e_794e_41c2_920e_2ff5156631da.slice/crio-ce1e5c4d7bef4e28aa3c99ed4045711aa5e91eab2a7e22df4ab2553c3dd5acd4 WatchSource:0}: Error finding container ce1e5c4d7bef4e28aa3c99ed4045711aa5e91eab2a7e22df4ab2553c3dd5acd4: Status 404 returned error can't find the container with id ce1e5c4d7bef4e28aa3c99ed4045711aa5e91eab2a7e22df4ab2553c3dd5acd4 Sep 15 12:26:25.540785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.540752 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4xr84\"" Sep 15 12:26:25.549199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.549175 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:26:25.558507 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.558483 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-d2lz7\"" Sep 15 12:26:25.567259 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.567236 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" Sep 15 12:26:25.601644 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.601189 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pzg8w\"" Sep 15 12:26:25.609853 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.609532 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mtkjn" Sep 15 12:26:25.821190 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.821153 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bd8db4dc9-m8kbs"] Sep 15 12:26:25.823963 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:25.823922 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80321730_1357_442e_9eb9_c72ce4cbf85b.slice/crio-9e182118d26fa2b08e714d543a96649d632a5aecf1a264d2c2d47ec7cc4d888a WatchSource:0}: Error finding container 9e182118d26fa2b08e714d543a96649d632a5aecf1a264d2c2d47ec7cc4d888a: Status 404 returned error can't find the container with id 9e182118d26fa2b08e714d543a96649d632a5aecf1a264d2c2d47ec7cc4d888a Sep 15 12:26:25.842882 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.842855 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b"] Sep 15 12:26:25.886351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:25.886321 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mtkjn"] Sep 15 12:26:25.890143 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:25.890119 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f55c5a_30e5_465e_a207_67dd23c5523a.slice/crio-459f7cf1e4107f80b70c3ce540790b3dc2d80d810b8e6be637ca1eaa2f8364fb WatchSource:0}: Error finding container 459f7cf1e4107f80b70c3ce540790b3dc2d80d810b8e6be637ca1eaa2f8364fb: Status 404 returned error can't find the container with id 459f7cf1e4107f80b70c3ce540790b3dc2d80d810b8e6be637ca1eaa2f8364fb Sep 15 12:26:26.482726 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.482681 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw2fs" event={"ID":"f4a9b48e-794e-41c2-920e-2ff5156631da","Type":"ContainerStarted","Data":"ce1e5c4d7bef4e28aa3c99ed4045711aa5e91eab2a7e22df4ab2553c3dd5acd4"} Sep 15 12:26:26.484295 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.484266 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" event={"ID":"f0996384-ef27-42a2-b215-daa5403e27b4","Type":"ContainerStarted","Data":"375937573dbceac01175a6d9d635e1be518849af75241ce5e9357f2d1fc2efc6"} Sep 15 12:26:26.486428 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.486400 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dr79g" event={"ID":"e1339667-307a-4bfd-a75c-a9d3e8d81e2d","Type":"ContainerStarted","Data":"3e42049accd4aa490fae59592798297f12955cf35ce0b3c247f8406a8536ad73"} Sep 15 12:26:26.486579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.486448 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dr79g" event={"ID":"e1339667-307a-4bfd-a75c-a9d3e8d81e2d","Type":"ContainerStarted","Data":"30f093bbca81adf4d73a3c52663eaa2e454479a129a7acb308ab7c850ab8de98"} Sep 15 12:26:26.486639 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.486624 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:26:26.488584 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.488560 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtkjn" event={"ID":"00f55c5a-30e5-465e-a207-67dd23c5523a","Type":"ContainerStarted","Data":"459f7cf1e4107f80b70c3ce540790b3dc2d80d810b8e6be637ca1eaa2f8364fb"} Sep 15 12:26:26.490973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.490946 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" event={"ID":"80321730-1357-442e-9eb9-c72ce4cbf85b","Type":"ContainerStarted","Data":"e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43"} Sep 15 12:26:26.491078 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.490990 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" event={"ID":"80321730-1357-442e-9eb9-c72ce4cbf85b","Type":"ContainerStarted","Data":"9e182118d26fa2b08e714d543a96649d632a5aecf1a264d2c2d47ec7cc4d888a"} Sep 15 12:26:26.491139 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.491126 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:26:26.503973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.503356 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dr79g" podStartSLOduration=66.503332776 podStartE2EDuration="1m6.503332776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:26.502502613 +0000 UTC m=+67.857348942" watchObservedRunningTime="2026-09-15 12:26:26.503332776 +0000 UTC m=+67.858179102" Sep 15 12:26:26.523442 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:26.523357 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" podStartSLOduration=66.523337678 podStartE2EDuration="1m6.523337678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:26.522667721 +0000 UTC m=+67.877514045" watchObservedRunningTime="2026-09-15 12:26:26.523337678 +0000 UTC m=+67.878183996" Sep 15 12:26:27.499313 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:27.499227 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw2fs" event={"ID":"f4a9b48e-794e-41c2-920e-2ff5156631da","Type":"ContainerStarted","Data":"bba1577d78e8afd318b770c7bce52f8249f37941a7124409ea9d110eb87ca15f"} Sep 15 12:26:28.184429 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.184393 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5d9dd5889b-hk9s6"] Sep 15 12:26:28.190919 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.190882 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-lzq77"] Sep 15 12:26:28.192071 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.191682 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.194058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.194039 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk"] Sep 15 12:26:28.194177 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.194162 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lzq77" Sep 15 12:26:28.195480 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.194865 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 15 12:26:28.195480 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.195079 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 15 12:26:28.195480 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.195297 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 12:26:28.195688 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.195567 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2qk5h\"" Sep 15 12:26:28.195865 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.195848 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 15 12:26:28.196362 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.195863 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 12:26:28.196362 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.195888 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 15 12:26:28.198356 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.197573 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 15 12:26:28.198356 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.197887 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.198356 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.198079 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 15 12:26:28.198356 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.198276 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dg79v\"" Sep 15 12:26:28.200733 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.200494 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 15 12:26:28.200733 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.200566 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xxd65\"" Sep 15 12:26:28.200889 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.200766 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 12:26:28.201889 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.201747 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 15 12:26:28.202353 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.201958 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 12:26:28.205278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.205122 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lzq77"] Sep 15 12:26:28.205461 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.205430 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk"] Sep 15 12:26:28.207981 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.207962 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d9dd5889b-hk9s6"] Sep 15 12:26:28.228950 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.228922 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvvlw\" (UniqueName: \"kubernetes.io/projected/25a3f5bc-8afb-46b8-8ea4-0acf695bfc92-kube-api-access-hvvlw\") pod \"downloads-76f99584f4-lzq77\" (UID: \"25a3f5bc-8afb-46b8-8ea4-0acf695bfc92\") " pod="openshift-console/downloads-76f99584f4-lzq77" Sep 15 12:26:28.229093 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.228976 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rrf\" (UniqueName: \"kubernetes.io/projected/87e210b6-091d-42a9-8d83-403afff524e7-kube-api-access-p8rrf\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.229093 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.229010 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2fbef136-c8c8-4f1e-8226-60ee42734392-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.229093 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.229047 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-default-certificate\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.229093 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.229071 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2fbef136-c8c8-4f1e-8226-60ee42734392-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.229309 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.229154 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-metrics-certs\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.229309 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.229189 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e210b6-091d-42a9-8d83-403afff524e7-service-ca-bundle\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.229309 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.229213 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-stats-auth\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.229490 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.229320 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lhn\" (UniqueName: \"kubernetes.io/projected/2fbef136-c8c8-4f1e-8226-60ee42734392-kube-api-access-w4lhn\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.330578 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330533 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-default-certificate\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.330578 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330579 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2fbef136-c8c8-4f1e-8226-60ee42734392-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.330794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330638 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-metrics-certs\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.330794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330679 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e210b6-091d-42a9-8d83-403afff524e7-service-ca-bundle\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.330794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330705 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-stats-auth\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.330794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330741 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lhn\" (UniqueName: \"kubernetes.io/projected/2fbef136-c8c8-4f1e-8226-60ee42734392-kube-api-access-w4lhn\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.330794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330786 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvvlw\" (UniqueName: \"kubernetes.io/projected/25a3f5bc-8afb-46b8-8ea4-0acf695bfc92-kube-api-access-hvvlw\") pod \"downloads-76f99584f4-lzq77\" (UID: \"25a3f5bc-8afb-46b8-8ea4-0acf695bfc92\") " pod="openshift-console/downloads-76f99584f4-lzq77" Sep 15 12:26:28.331152 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330833 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rrf\" (UniqueName: \"kubernetes.io/projected/87e210b6-091d-42a9-8d83-403afff524e7-kube-api-access-p8rrf\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.331152 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.330863 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2fbef136-c8c8-4f1e-8226-60ee42734392-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.332204 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.332146 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87e210b6-091d-42a9-8d83-403afff524e7-service-ca-bundle\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.332681 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.332620 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2fbef136-c8c8-4f1e-8226-60ee42734392-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.334435 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.334363 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-metrics-certs\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.334534 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.334505 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2fbef136-c8c8-4f1e-8226-60ee42734392-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.334800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.334760 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-stats-auth\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.335255 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.335236 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/87e210b6-091d-42a9-8d83-403afff524e7-default-certificate\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.343152 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.342717 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rrf\" (UniqueName: \"kubernetes.io/projected/87e210b6-091d-42a9-8d83-403afff524e7-kube-api-access-p8rrf\") pod \"router-default-5d9dd5889b-hk9s6\" (UID: \"87e210b6-091d-42a9-8d83-403afff524e7\") " pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.343217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.343193 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvvlw\" (UniqueName: \"kubernetes.io/projected/25a3f5bc-8afb-46b8-8ea4-0acf695bfc92-kube-api-access-hvvlw\") pod \"downloads-76f99584f4-lzq77\" (UID: \"25a3f5bc-8afb-46b8-8ea4-0acf695bfc92\") " pod="openshift-console/downloads-76f99584f4-lzq77" Sep 15 12:26:28.344659 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.344628 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lhn\" (UniqueName: \"kubernetes.io/projected/2fbef136-c8c8-4f1e-8226-60ee42734392-kube-api-access-w4lhn\") pod \"cluster-monitoring-operator-6cd954b897-wbvdk\" (UID: \"2fbef136-c8c8-4f1e-8226-60ee42734392\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.512000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.511118 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:28.515397 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.515339 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bw2fs" event={"ID":"f4a9b48e-794e-41c2-920e-2ff5156631da","Type":"ContainerStarted","Data":"d507d59dfbcfc5320af137931235ca3060b674e76ee68df4aa52fefe678413f5"} Sep 15 12:26:28.519061 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.518998 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" event={"ID":"f0996384-ef27-42a2-b215-daa5403e27b4","Type":"ContainerStarted","Data":"ec794adc26424ed267bce9198e41707bbedbf5f90aee1a0c70c05b07860449d2"} Sep 15 12:26:28.519187 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.519064 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lzq77" Sep 15 12:26:28.522107 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.522013 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtkjn" event={"ID":"00f55c5a-30e5-465e-a207-67dd23c5523a","Type":"ContainerStarted","Data":"dd2d3743f43db23caabd64641d72d7619861ec5d10e393ab8efe849873906cf4"} Sep 15 12:26:28.527060 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.527037 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" Sep 15 12:26:28.531366 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:28.531265 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bw2fs" podStartSLOduration=66.989097148 podStartE2EDuration="1m8.531244963s" totalImagesPullingTime="1.542147815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:28.530429096 +0000 UTC m=+69.885275419" watchObservedRunningTime="2026-09-15 12:26:28.531244963 +0000 UTC m=+69.886091286" Sep 15 12:26:29.528248 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:29.528210 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" event={"ID":"f0996384-ef27-42a2-b215-daa5403e27b4","Type":"ContainerStarted","Data":"fdc832ec4b00a6b5752b8bcde43bbdb9638cd4d127da5868b903f1d0c1e23f90"} Sep 15 12:26:29.530348 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:29.530307 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mtkjn" event={"ID":"00f55c5a-30e5-465e-a207-67dd23c5523a","Type":"ContainerStarted","Data":"18bbd3e812d64f876581a12de1d2e37a511110a5ca4a2bca494637e7435c7fa2"} Sep 15 12:26:29.546443 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:29.546365 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-5ct2b" podStartSLOduration=63.084036464 podStartE2EDuration="1m5.546348027s" totalImagesPullingTime="2.462311563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:29.545167717 +0000 UTC m=+70.900014051" watchObservedRunningTime="2026-09-15 12:26:29.546348027 +0000 UTC m=+70.901194348" Sep 15 12:26:29.578320 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:29.578251 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mtkjn" podStartSLOduration=34.132701092 podStartE2EDuration="36.578216591s" totalImagesPullingTime="2.445515499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:29.577111907 +0000 UTC m=+70.931958243" watchObservedRunningTime="2026-09-15 12:26:29.578216591 +0000 UTC m=+70.933062915" Sep 15 12:26:30.243587 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.243553 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk"] Sep 15 12:26:30.247956 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:30.247933 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbef136_c8c8_4f1e_8226_60ee42734392.slice/crio-94365c32aea77e559780218e32213ce6704fa3224c6eaafaebcaa72e6500ba92 WatchSource:0}: Error finding container 94365c32aea77e559780218e32213ce6704fa3224c6eaafaebcaa72e6500ba92: Status 404 returned error can't find the container with id 94365c32aea77e559780218e32213ce6704fa3224c6eaafaebcaa72e6500ba92 Sep 15 12:26:30.257158 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.257127 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lzq77"] Sep 15 12:26:30.260157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.260135 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5d9dd5889b-hk9s6"] Sep 15 12:26:30.262930 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:30.262904 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e210b6_091d_42a9_8d83_403afff524e7.slice/crio-e0b929be10db1ea3ab52d8a2a1460d77e2e7dcb0c92eb2513b9e89b80e64dac0 WatchSource:0}: Error finding container e0b929be10db1ea3ab52d8a2a1460d77e2e7dcb0c92eb2513b9e89b80e64dac0: Status 404 returned error can't find the container with id e0b929be10db1ea3ab52d8a2a1460d77e2e7dcb0c92eb2513b9e89b80e64dac0 Sep 15 12:26:30.263880 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:30.263855 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a3f5bc_8afb_46b8_8ea4_0acf695bfc92.slice/crio-2943d5748d40a9ff144763574c67bbeeb136530c5db67d052eb62252b9ef34f1 WatchSource:0}: Error finding container 2943d5748d40a9ff144763574c67bbeeb136530c5db67d052eb62252b9ef34f1: Status 404 returned error can't find the container with id 2943d5748d40a9ff144763574c67bbeeb136530c5db67d052eb62252b9ef34f1 Sep 15 12:26:30.535432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.535334 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" event={"ID":"2fbef136-c8c8-4f1e-8226-60ee42734392","Type":"ContainerStarted","Data":"94365c32aea77e559780218e32213ce6704fa3224c6eaafaebcaa72e6500ba92"} Sep 15 12:26:30.536981 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.536946 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" event={"ID":"87e210b6-091d-42a9-8d83-403afff524e7","Type":"ContainerStarted","Data":"f377ab68592e48bdda41fa4e167d6674703f4d3aeb12238c3e9a93c2433023cb"} Sep 15 12:26:30.537122 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.537002 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" event={"ID":"87e210b6-091d-42a9-8d83-403afff524e7","Type":"ContainerStarted","Data":"e0b929be10db1ea3ab52d8a2a1460d77e2e7dcb0c92eb2513b9e89b80e64dac0"} Sep 15 12:26:30.538604 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.538578 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-px8td" event={"ID":"ea8d9432-e366-40c2-81fd-b478d05a1df7","Type":"ContainerStarted","Data":"39300b7cd0b1367ca2df3c79a5875a38e0f5d2a0811418c228c837b40d189803"} Sep 15 12:26:30.539829 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.539802 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lzq77" event={"ID":"25a3f5bc-8afb-46b8-8ea4-0acf695bfc92","Type":"ContainerStarted","Data":"2943d5748d40a9ff144763574c67bbeeb136530c5db67d052eb62252b9ef34f1"} Sep 15 12:26:30.540198 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.540180 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mtkjn" Sep 15 12:26:30.555960 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.555905 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" podStartSLOduration=2.555888726 podStartE2EDuration="2.555888726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:30.554605462 +0000 UTC m=+71.909451809" watchObservedRunningTime="2026-09-15 12:26:30.555888726 +0000 UTC m=+71.910735074" Sep 15 12:26:30.568204 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:30.568156 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-px8td" podStartSLOduration=65.765410576 podStartE2EDuration="1m10.568137609s" totalImagesPullingTime="4.802727033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:30.567464671 +0000 UTC m=+71.922310998" watchObservedRunningTime="2026-09-15 12:26:30.568137609 +0000 UTC m=+71.922983957" Sep 15 12:26:31.127394 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.127336 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wwn96"] Sep 15 12:26:31.163306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.163270 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wwn96"] Sep 15 12:26:31.163507 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.163458 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.167044 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.167019 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9kbt\"" Sep 15 12:26:31.167950 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.167920 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 12:26:31.168074 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.167954 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 12:26:31.168074 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.167967 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 12:26:31.168074 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.167971 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 12:26:31.253443 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.253402 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m5bp\" (UniqueName: \"kubernetes.io/projected/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-api-access-2m5bp\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.253641 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.253458 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0f7776b-cd11-4862-9b0c-e638e096aaa5-crio-socket\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.253641 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.253510 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0f7776b-cd11-4862-9b0c-e638e096aaa5-data-volume\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.253641 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.253598 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.253641 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.253632 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.354738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.354696 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m5bp\" (UniqueName: \"kubernetes.io/projected/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-api-access-2m5bp\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.354738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.354748 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0f7776b-cd11-4862-9b0c-e638e096aaa5-crio-socket\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.354991 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.354843 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e0f7776b-cd11-4862-9b0c-e638e096aaa5-crio-socket\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.354991 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.354871 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:31.354991 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.354934 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0f7776b-cd11-4862-9b0c-e638e096aaa5-data-volume\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.354991 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.354987 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.355208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.355023 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.355208 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.355164 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 12:26:31.355309 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.355233 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:31.855213785 +0000 UTC m=+73.210060092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : secret "insights-runtime-extractor-tls" not found Sep 15 12:26:31.355309 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.355284 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 12:26:31.355450 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.355350 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:31.855328817 +0000 UTC m=+73.210175134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : configmap "kube-rbac-proxy" not found Sep 15 12:26:31.355450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.355421 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0f7776b-cd11-4862-9b0c-e638e096aaa5-data-volume\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.357921 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.357900 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/373725c3-1275-4adf-99dc-6a3cf09e620b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-jv74g\" (UID: \"373725c3-1275-4adf-99dc-6a3cf09e620b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:31.366567 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.366538 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m5bp\" (UniqueName: \"kubernetes.io/projected/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-api-access-2m5bp\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.511754 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.511548 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:31.514731 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.514707 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:31.548768 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.548722 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:31.549983 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.549959 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5d9dd5889b-hk9s6" Sep 15 12:26:31.595201 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.595003 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-928zd\"" Sep 15 12:26:31.602279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.602114 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" Sep 15 12:26:31.859725 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.859634 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.859725 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.859698 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:31.859941 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.859737 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 12:26:31.859941 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.859802 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 12:26:31.859941 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.859823 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:32.859801765 +0000 UTC m=+74.214648082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : configmap "kube-rbac-proxy" not found Sep 15 12:26:31.859941 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:31.859842 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:32.859831544 +0000 UTC m=+74.214677859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : secret "insights-runtime-extractor-tls" not found Sep 15 12:26:31.918813 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:31.918720 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g"] Sep 15 12:26:32.120398 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.120291 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtkjn_00f55c5a-30e5-465e-a207-67dd23c5523a/dns/0.log" Sep 15 12:26:32.294546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.294508 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtkjn_00f55c5a-30e5-465e-a207-67dd23c5523a/kube-rbac-proxy/0.log" Sep 15 12:26:32.445555 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:32.445472 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373725c3_1275_4adf_99dc_6a3cf09e620b.slice/crio-7324bcf2b4f860013266bcb888b6ffc955ba46971560256c3768df5f6ddb0f65 WatchSource:0}: Error finding container 7324bcf2b4f860013266bcb888b6ffc955ba46971560256c3768df5f6ddb0f65: Status 404 returned error can't find the container with id 7324bcf2b4f860013266bcb888b6ffc955ba46971560256c3768df5f6ddb0f65 Sep 15 12:26:32.552683 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.552645 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" event={"ID":"373725c3-1275-4adf-99dc-6a3cf09e620b","Type":"ContainerStarted","Data":"7324bcf2b4f860013266bcb888b6ffc955ba46971560256c3768df5f6ddb0f65"} Sep 15 12:26:32.869604 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.869571 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:32.869604 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.869619 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:32.869840 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:32.869714 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 12:26:32.869840 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:32.869802 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:34.869780561 +0000 UTC m=+76.224626876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : configmap "kube-rbac-proxy" not found Sep 15 12:26:32.869840 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:32.869807 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 12:26:32.869977 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:32.869871 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:34.86985394 +0000 UTC m=+76.224700245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : secret "insights-runtime-extractor-tls" not found Sep 15 12:26:32.875329 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.875256 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c7f465b9d-skxb6"] Sep 15 12:26:32.893168 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.893134 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c7f465b9d-skxb6"] Sep 15 12:26:32.893312 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.893254 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:32.895495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.895467 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pkp8g\"" Sep 15 12:26:32.895620 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.895534 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 15 12:26:32.895759 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.895659 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 15 12:26:32.895844 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.895759 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 15 12:26:32.896306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.896284 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 15 12:26:32.896662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.896643 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 15 12:26:32.901493 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.901460 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 15 12:26:32.970855 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.970807 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-oauth-serving-cert\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:32.971040 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.970941 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-config\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:32.971040 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.970989 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-trusted-ca-bundle\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:32.971040 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.971020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29rj\" (UniqueName: \"kubernetes.io/projected/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-kube-api-access-x29rj\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:32.971214 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.971069 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-serving-cert\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:32.971214 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.971093 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-service-ca\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:32.971214 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:32.971162 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-oauth-config\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.073266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.072267 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-oauth-config\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.073266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.072344 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-oauth-serving-cert\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.073266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.072417 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-config\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.073266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.072447 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-trusted-ca-bundle\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.073266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.072472 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29rj\" (UniqueName: \"kubernetes.io/projected/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-kube-api-access-x29rj\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.073266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.072514 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-serving-cert\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.073266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.072540 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-service-ca\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.074103 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.073759 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-service-ca\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.074103 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.073937 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-config\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.074103 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.074039 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-oauth-serving-cert\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.074349 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.074329 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-trusted-ca-bundle\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.075895 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.075238 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-oauth-config\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.076633 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.076592 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-serving-cert\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.083470 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.083414 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29rj\" (UniqueName: \"kubernetes.io/projected/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-kube-api-access-x29rj\") pod \"console-5c7f465b9d-skxb6\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.094422 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.094226 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxrzq_d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55/dns-node-resolver/0.log" Sep 15 12:26:33.206126 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.205881 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:33.541280 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.541246 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c7f465b9d-skxb6"] Sep 15 12:26:33.560083 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.560036 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" event={"ID":"2fbef136-c8c8-4f1e-8226-60ee42734392","Type":"ContainerStarted","Data":"e57258e235ed263a82d1e4e14abf60bd9402f4c92a996d11656fc4c85a95a435"} Sep 15 12:26:33.577683 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.577618 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wbvdk" podStartSLOduration=3.05831878 podStartE2EDuration="5.577594793s" totalImagesPullingTime="2.519276013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:33.575665265 +0000 UTC m=+74.930511589" watchObservedRunningTime="2026-09-15 12:26:33.577594793 +0000 UTC m=+74.932441117" Sep 15 12:26:33.691934 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:33.691902 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6bd8db4dc9-m8kbs_80321730-1357-442e-9eb9-c72ce4cbf85b/registry/0.log" Sep 15 12:26:33.929774 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:33.929692 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9d15ec_ec26_46bd_9605_d0739e2c5f34.slice/crio-72a3eaa5e56752bdadf25aa400a6e969ff6a58fa9f54cb5e669658dd63f3c6da WatchSource:0}: Error finding container 72a3eaa5e56752bdadf25aa400a6e969ff6a58fa9f54cb5e669658dd63f3c6da: Status 404 returned error can't find the container with id 72a3eaa5e56752bdadf25aa400a6e969ff6a58fa9f54cb5e669658dd63f3c6da Sep 15 12:26:34.491643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:34.491526 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnvms_259030fd-b0aa-4c90-b2b6-b3a82c8e994c/node-ca/0.log" Sep 15 12:26:34.566403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:34.566183 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" event={"ID":"373725c3-1275-4adf-99dc-6a3cf09e620b","Type":"ContainerStarted","Data":"d32be8e509db7b6e0232258d08eac88e9889a7fb96867354bb342f2804f99afa"} Sep 15 12:26:34.568327 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:34.568290 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7f465b9d-skxb6" event={"ID":"7f9d15ec-ec26-46bd-9605-d0739e2c5f34","Type":"ContainerStarted","Data":"72a3eaa5e56752bdadf25aa400a6e969ff6a58fa9f54cb5e669658dd63f3c6da"} Sep 15 12:26:34.582531 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:34.582449 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-jv74g" podStartSLOduration=33.761946146 podStartE2EDuration="35.582432741s" totalImagesPullingTime="1.820486595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:34.580919499 +0000 UTC m=+75.935765824" watchObservedRunningTime="2026-09-15 12:26:34.582432741 +0000 UTC m=+75.937279063" Sep 15 12:26:34.692691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:34.692656 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d9dd5889b-hk9s6_87e210b6-091d-42a9-8d83-403afff524e7/router/0.log" Sep 15 12:26:34.890565 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:34.890473 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:34.890565 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:34.890532 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:34.890769 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:34.890608 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 12:26:34.890769 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:34.890649 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 12:26:34.890769 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:34.890695 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:38.890672466 +0000 UTC m=+80.245518771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : configmap "kube-rbac-proxy" not found Sep 15 12:26:34.890769 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:34.890715 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:38.890705204 +0000 UTC m=+80.245551511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : secret "insights-runtime-extractor-tls" not found Sep 15 12:26:35.092687 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:35.092596 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ct9mg_b512fd0d-b99f-4f92-83f9-a7ace0c64f65/serve-healthcheck-canary/0.log" Sep 15 12:26:37.397084 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.397049 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p"] Sep 15 12:26:37.418397 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.418353 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p"] Sep 15 12:26:37.418554 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.418492 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.420983 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.420955 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 12:26:37.421172 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.421158 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 15 12:26:37.421353 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.421337 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 15 12:26:37.421817 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.421798 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cx4f2\"" Sep 15 12:26:37.512702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.512668 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.512832 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.512761 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc104f-ef60-4779-9dae-b9caf4682c6e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.512832 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.512805 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6g8m\" (UniqueName: \"kubernetes.io/projected/bfdc104f-ef60-4779-9dae-b9caf4682c6e-kube-api-access-z6g8m\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.512940 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.512887 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.584782 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.584733 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7f465b9d-skxb6" event={"ID":"7f9d15ec-ec26-46bd-9605-d0739e2c5f34","Type":"ContainerStarted","Data":"dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82"} Sep 15 12:26:37.603997 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.603938 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c7f465b9d-skxb6" podStartSLOduration=2.070627902 podStartE2EDuration="5.603914539s" totalImagesPullingTime="3.533286637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:37.602197795 +0000 UTC m=+78.957044120" watchObservedRunningTime="2026-09-15 12:26:37.603914539 +0000 UTC m=+78.958760862" Sep 15 12:26:37.613779 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.613740 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.613945 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.613810 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc104f-ef60-4779-9dae-b9caf4682c6e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.613945 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.613841 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6g8m\" (UniqueName: \"kubernetes.io/projected/bfdc104f-ef60-4779-9dae-b9caf4682c6e-kube-api-access-z6g8m\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.613945 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:37.613852 2550 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 15 12:26:37.613945 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.613883 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.613945 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:37.613920 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-tls podName:bfdc104f-ef60-4779-9dae-b9caf4682c6e nodeName:}" failed. No retries permitted until 2026-09-15 12:26:38.113900189 +0000 UTC m=+79.468746495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-8tp4p" (UID: "bfdc104f-ef60-4779-9dae-b9caf4682c6e") : secret "prometheus-operator-tls" not found Sep 15 12:26:37.614723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.614701 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc104f-ef60-4779-9dae-b9caf4682c6e-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.617020 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.616975 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:37.621810 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:37.621784 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6g8m\" (UniqueName: \"kubernetes.io/projected/bfdc104f-ef60-4779-9dae-b9caf4682c6e-kube-api-access-z6g8m\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:38.118179 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:38.118138 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:38.121033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:38.121001 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfdc104f-ef60-4779-9dae-b9caf4682c6e-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8tp4p\" (UID: \"bfdc104f-ef60-4779-9dae-b9caf4682c6e\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:38.331900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:38.331863 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" Sep 15 12:26:38.608304 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:38.608269 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p"] Sep 15 12:26:38.611557 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:38.611517 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdc104f_ef60_4779_9dae_b9caf4682c6e.slice/crio-1cd57e013f2a739827b66511c8deb5e72f42178d7ccc61351acac2075ab9e455 WatchSource:0}: Error finding container 1cd57e013f2a739827b66511c8deb5e72f42178d7ccc61351acac2075ab9e455: Status 404 returned error can't find the container with id 1cd57e013f2a739827b66511c8deb5e72f42178d7ccc61351acac2075ab9e455 Sep 15 12:26:38.926362 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:38.926320 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:38.926567 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:38.926395 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:38.926567 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:38.926465 2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 12:26:38.926567 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:38.926558 2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 12:26:38.926735 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:38.926562 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:46.926526177 +0000 UTC m=+88.281372486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : configmap "kube-rbac-proxy" not found Sep 15 12:26:38.926735 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:38.926638 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls podName:e0f7776b-cd11-4862-9b0c-e638e096aaa5 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:46.926608888 +0000 UTC m=+88.281455192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wwn96" (UID: "e0f7776b-cd11-4862-9b0c-e638e096aaa5") : secret "insights-runtime-extractor-tls" not found Sep 15 12:26:39.594953 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:39.594910 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" event={"ID":"bfdc104f-ef60-4779-9dae-b9caf4682c6e","Type":"ContainerStarted","Data":"1cd57e013f2a739827b66511c8deb5e72f42178d7ccc61351acac2075ab9e455"} Sep 15 12:26:40.549885 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:40.549849 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mtkjn" Sep 15 12:26:40.601449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:40.601408 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" event={"ID":"bfdc104f-ef60-4779-9dae-b9caf4682c6e","Type":"ContainerStarted","Data":"c170f4ba18bc3f18fe869d7374927c66336d44df7fdc0eae6e2c819469970f64"} Sep 15 12:26:40.601636 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:40.601478 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" event={"ID":"bfdc104f-ef60-4779-9dae-b9caf4682c6e","Type":"ContainerStarted","Data":"e32c7f5159e1e4aafaa5872200c315e704fb5995734794c738aae9fdd2011e16"} Sep 15 12:26:40.619073 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:40.619004 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8tp4p" podStartSLOduration=2.261866984 podStartE2EDuration="3.618980663s" totalImagesPullingTime="1.357113679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:40.618158044 +0000 UTC m=+81.973004371" watchObservedRunningTime="2026-09-15 12:26:40.618980663 +0000 UTC m=+81.973826989" Sep 15 12:26:42.773807 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.773666 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bmwfb"] Sep 15 12:26:42.778626 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.778597 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.781325 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.781294 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wzdhx\"" Sep 15 12:26:42.781474 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.781299 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 12:26:42.785270 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.782601 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 12:26:42.785270 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.782835 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 12:26:42.865925 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.865883 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.865939 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mfn\" (UniqueName: \"kubernetes.io/projected/7ca9ce81-ca99-41f1-b380-f621b12b6604-kube-api-access-28mfn\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.866003 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-tls\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.866032 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ca9ce81-ca99-41f1-b380-f621b12b6604-metrics-client-ca\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.866060 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-accelerators-collector-config\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866277 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.866152 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-root\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866277 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.866200 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-textfile\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.866293 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-sys\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.866369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.866322 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-wtmp\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967124 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-root\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967355 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967174 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-textfile\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967355 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967265 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-root\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967488 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967403 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-sys\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967488 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967457 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-wtmp\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967500 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967458 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-sys\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967536 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mfn\" (UniqueName: \"kubernetes.io/projected/7ca9ce81-ca99-41f1-b380-f621b12b6604-kube-api-access-28mfn\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967574 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-wtmp\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967752 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967561 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-textfile\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967752 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967682 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-tls\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967752 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967708 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ca9ce81-ca99-41f1-b380-f621b12b6604-metrics-client-ca\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967752 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.967737 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-accelerators-collector-config\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.967916 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:42.967845 2550 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 15 12:26:42.967953 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:42.967933 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-tls podName:7ca9ce81-ca99-41f1-b380-f621b12b6604 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:43.467911908 +0000 UTC m=+84.822758226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-tls") pod "node-exporter-bmwfb" (UID: "7ca9ce81-ca99-41f1-b380-f621b12b6604") : secret "node-exporter-tls" not found Sep 15 12:26:42.968263 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.968244 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ca9ce81-ca99-41f1-b380-f621b12b6604-metrics-client-ca\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.968363 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.968317 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-accelerators-collector-config\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.970102 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.970074 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:42.975696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:42.975637 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mfn\" (UniqueName: \"kubernetes.io/projected/7ca9ce81-ca99-41f1-b380-f621b12b6604-kube-api-access-28mfn\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:43.206921 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.206823 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:43.206921 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.206913 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:43.213228 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.213196 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:43.474129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.474029 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-tls\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:43.476770 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.476735 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ca9ce81-ca99-41f1-b380-f621b12b6604-node-exporter-tls\") pod \"node-exporter-bmwfb\" (UID: \"7ca9ce81-ca99-41f1-b380-f621b12b6604\") " pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:43.618960 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.618932 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:26:43.697246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.697211 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bmwfb" Sep 15 12:26:43.849723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.849630 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 12:26:43.854188 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.854156 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.857142 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.856924 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 15 12:26:43.858208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.858018 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 15 12:26:43.858630 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.858509 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 15 12:26:43.858630 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.858101 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 15 12:26:43.858903 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.858866 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 15 12:26:43.859339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.859001 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 15 12:26:43.859339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.858050 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pvgrd\"" Sep 15 12:26:43.860222 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.859759 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 15 12:26:43.860222 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.859948 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 15 12:26:43.860334 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.860255 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 15 12:26:43.868200 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.868173 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 12:26:43.978912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.978875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.978912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.978918 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.978945 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.978985 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979019 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979046 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-web-config\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979075 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d96fad9-6b46-462f-81f9-4729c93ed38c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979109 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d96fad9-6b46-462f-81f9-4729c93ed38c-config-out\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztvf5\" (UniqueName: \"kubernetes.io/projected/7d96fad9-6b46-462f-81f9-4729c93ed38c-kube-api-access-ztvf5\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979201 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979243 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979287 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:43.979530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:43.979343 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080659 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080549 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d96fad9-6b46-462f-81f9-4729c93ed38c-config-out\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080659 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080606 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztvf5\" (UniqueName: \"kubernetes.io/projected/7d96fad9-6b46-462f-81f9-4729c93ed38c-kube-api-access-ztvf5\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080659 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080644 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080952 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080694 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080952 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080728 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080952 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080761 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080952 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080866 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080952 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080911 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.080952 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080938 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.081251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.080977 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.081251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.081010 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.081251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.081038 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-web-config\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.081251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.081082 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d96fad9-6b46-462f-81f9-4729c93ed38c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.081485 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:44.081466 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-trusted-ca-bundle podName:7d96fad9-6b46-462f-81f9-4729c93ed38c nodeName:}" failed. No retries permitted until 2026-09-15 12:26:44.581442947 +0000 UTC m=+85.936289266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7d96fad9-6b46-462f-81f9-4729c93ed38c") : configmap references non-existent config key: ca-bundle.crt Sep 15 12:26:44.081695 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.081549 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.081695 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.081661 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.085186 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.085119 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.085186 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.085163 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-web-config\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.086031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.086002 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.086339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.086313 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d96fad9-6b46-462f-81f9-4729c93ed38c-config-out\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.086998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.086836 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.087595 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.087573 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.087754 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.087732 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.088139 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.088115 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d96fad9-6b46-462f-81f9-4729c93ed38c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.088247 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.088159 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d96fad9-6b46-462f-81f9-4729c93ed38c-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.092546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.092516 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztvf5\" (UniqueName: \"kubernetes.io/projected/7d96fad9-6b46-462f-81f9-4729c93ed38c-kube-api-access-ztvf5\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.586208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.586159 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.587359 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.587324 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d96fad9-6b46-462f-81f9-4729c93ed38c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d96fad9-6b46-462f-81f9-4729c93ed38c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:44.772165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:44.772120 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 12:26:47.012894 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.012830 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:47.013432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.012902 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:47.013432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.013420 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e0f7776b-cd11-4862-9b0c-e638e096aaa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:47.015460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.015437 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e0f7776b-cd11-4862-9b0c-e638e096aaa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wwn96\" (UID: \"e0f7776b-cd11-4862-9b0c-e638e096aaa5\") " pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:47.076763 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.076727 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wwn96" Sep 15 12:26:47.379649 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:47.379595 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca9ce81_ca99_41f1_b380_f621b12b6604.slice/crio-884328149e6b44b366e110ee7a61d03a1c62d7197aab887b945b5100f1876c8a WatchSource:0}: Error finding container 884328149e6b44b366e110ee7a61d03a1c62d7197aab887b945b5100f1876c8a: Status 404 returned error can't find the container with id 884328149e6b44b366e110ee7a61d03a1c62d7197aab887b945b5100f1876c8a Sep 15 12:26:47.507263 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.507230 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:26:47.544474 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.544360 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9"] Sep 15 12:26:47.548143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.548116 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:47.550674 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.550493 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 15 12:26:47.550827 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.550760 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zv7fb\"" Sep 15 12:26:47.556761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.556724 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9"] Sep 15 12:26:47.617611 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.617576 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3cc89b7f-ef8a-496d-9d6f-296856a2a966-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4hqf9\" (UID: \"3cc89b7f-ef8a-496d-9d6f-296856a2a966\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:47.617816 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.617792 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmk4b\" (UniqueName: \"kubernetes.io/projected/3cc89b7f-ef8a-496d-9d6f-296856a2a966-kube-api-access-gmk4b\") pod \"monitoring-plugin-5d85778b9d-4hqf9\" (UID: \"3cc89b7f-ef8a-496d-9d6f-296856a2a966\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:47.635235 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.635106 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lzq77" event={"ID":"25a3f5bc-8afb-46b8-8ea4-0acf695bfc92","Type":"ContainerStarted","Data":"a03fff42035940738ef4c942cc31e05699a2bc30c3550a798a5a4c1384056324"} Sep 15 12:26:47.635572 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.635509 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-lzq77" Sep 15 12:26:47.637023 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.636995 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmwfb" event={"ID":"7ca9ce81-ca99-41f1-b380-f621b12b6604","Type":"ContainerStarted","Data":"884328149e6b44b366e110ee7a61d03a1c62d7197aab887b945b5100f1876c8a"} Sep 15 12:26:47.637212 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.637185 2550 patch_prober.go:28] interesting pod/downloads-76f99584f4-lzq77 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.17:8080/\": dial tcp 10.132.0.17:8080: connect: connection refused" start-of-body= Sep 15 12:26:47.637287 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.637242 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-lzq77" podUID="25a3f5bc-8afb-46b8-8ea4-0acf695bfc92" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.17:8080/\": dial tcp 10.132.0.17:8080: connect: connection refused" Sep 15 12:26:47.652883 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.652818 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-lzq77" podStartSLOduration=2.454148457 podStartE2EDuration="19.652801493s" totalImagesPullingTime="17.198653036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:47.651152715 +0000 UTC m=+89.005999038" watchObservedRunningTime="2026-09-15 12:26:47.652801493 +0000 UTC m=+89.007647858" Sep 15 12:26:47.656322 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.656233 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wwn96"] Sep 15 12:26:47.659954 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:47.659909 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f7776b_cd11_4862_9b0c_e638e096aaa5.slice/crio-56055675c5a96d0d8bada6fd241417036218d348bc2d561a186f7fbc185965cc WatchSource:0}: Error finding container 56055675c5a96d0d8bada6fd241417036218d348bc2d561a186f7fbc185965cc: Status 404 returned error can't find the container with id 56055675c5a96d0d8bada6fd241417036218d348bc2d561a186f7fbc185965cc Sep 15 12:26:47.664852 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.664824 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 12:26:47.669157 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:47.669124 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d96fad9_6b46_462f_81f9_4729c93ed38c.slice/crio-04db2c4a3f0d738957e21b39669b13de5c45786a188ae1f1637a1ad9a1ca73fb WatchSource:0}: Error finding container 04db2c4a3f0d738957e21b39669b13de5c45786a188ae1f1637a1ad9a1ca73fb: Status 404 returned error can't find the container with id 04db2c4a3f0d738957e21b39669b13de5c45786a188ae1f1637a1ad9a1ca73fb Sep 15 12:26:47.719256 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.718970 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3cc89b7f-ef8a-496d-9d6f-296856a2a966-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4hqf9\" (UID: \"3cc89b7f-ef8a-496d-9d6f-296856a2a966\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:47.719256 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.719068 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmk4b\" (UniqueName: \"kubernetes.io/projected/3cc89b7f-ef8a-496d-9d6f-296856a2a966-kube-api-access-gmk4b\") pod \"monitoring-plugin-5d85778b9d-4hqf9\" (UID: \"3cc89b7f-ef8a-496d-9d6f-296856a2a966\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:47.719256 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:47.719154 2550 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 15 12:26:47.719256 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:26:47.719228 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc89b7f-ef8a-496d-9d6f-296856a2a966-monitoring-plugin-cert podName:3cc89b7f-ef8a-496d-9d6f-296856a2a966 nodeName:}" failed. No retries permitted until 2026-09-15 12:26:48.219208146 +0000 UTC m=+89.574054450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3cc89b7f-ef8a-496d-9d6f-296856a2a966-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-4hqf9" (UID: "3cc89b7f-ef8a-496d-9d6f-296856a2a966") : secret "monitoring-plugin-cert" not found Sep 15 12:26:47.727915 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:47.727890 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmk4b\" (UniqueName: \"kubernetes.io/projected/3cc89b7f-ef8a-496d-9d6f-296856a2a966-kube-api-access-gmk4b\") pod \"monitoring-plugin-5d85778b9d-4hqf9\" (UID: \"3cc89b7f-ef8a-496d-9d6f-296856a2a966\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:48.226951 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.226411 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3cc89b7f-ef8a-496d-9d6f-296856a2a966-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4hqf9\" (UID: \"3cc89b7f-ef8a-496d-9d6f-296856a2a966\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:48.230998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.230931 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3cc89b7f-ef8a-496d-9d6f-296856a2a966-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-4hqf9\" (UID: \"3cc89b7f-ef8a-496d-9d6f-296856a2a966\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:48.461272 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.461235 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:48.650110 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.649904 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wwn96" event={"ID":"e0f7776b-cd11-4862-9b0c-e638e096aaa5","Type":"ContainerStarted","Data":"ae95870fe6bc4d92b662bc01af52f318b122d4aa8e0265bd97e6434eac1d6f1c"} Sep 15 12:26:48.650110 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.650009 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wwn96" event={"ID":"e0f7776b-cd11-4862-9b0c-e638e096aaa5","Type":"ContainerStarted","Data":"56055675c5a96d0d8bada6fd241417036218d348bc2d561a186f7fbc185965cc"} Sep 15 12:26:48.651813 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.651779 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerStarted","Data":"04db2c4a3f0d738957e21b39669b13de5c45786a188ae1f1637a1ad9a1ca73fb"} Sep 15 12:26:48.655876 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.655843 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-lzq77" Sep 15 12:26:48.695879 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.694072 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79bf4f6947-ptp6q"] Sep 15 12:26:48.717320 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.715937 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.718844 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.718718 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bf4f6947-ptp6q"] Sep 15 12:26:48.832930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.832537 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-oauth-config\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.832930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.832589 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-service-ca\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.832930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.832631 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-console-config\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.832930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.832666 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9gq6\" (UniqueName: \"kubernetes.io/projected/59097b15-f7a9-457d-81a1-2de081788281-kube-api-access-x9gq6\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.832930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.832695 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-serving-cert\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.832930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.832728 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-trusted-ca-bundle\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.832930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.832812 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-oauth-serving-cert\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.934077 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.933987 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9gq6\" (UniqueName: \"kubernetes.io/projected/59097b15-f7a9-457d-81a1-2de081788281-kube-api-access-x9gq6\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.934077 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.934031 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-serving-cert\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.934317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.934089 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-trusted-ca-bundle\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.934317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.934176 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-oauth-serving-cert\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.934317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.934229 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-oauth-config\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.934317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.934253 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-service-ca\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.934317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.934295 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-console-config\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.935339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.935253 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-oauth-serving-cert\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.935339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.935323 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-console-config\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.935580 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.935373 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-service-ca\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.939169 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.939012 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-trusted-ca-bundle\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.939169 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.939162 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-serving-cert\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.951403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.950455 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-oauth-config\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.957245 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.957183 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9gq6\" (UniqueName: \"kubernetes.io/projected/59097b15-f7a9-457d-81a1-2de081788281-kube-api-access-x9gq6\") pod \"console-79bf4f6947-ptp6q\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:48.986060 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:48.985944 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9"] Sep 15 12:26:49.038808 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:49.038770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:49.566794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:49.566753 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bf4f6947-ptp6q"] Sep 15 12:26:49.592418 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:26:49.592359 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59097b15_f7a9_457d_81a1_2de081788281.slice/crio-d20f813482abd24cd54f20e426dcfdbbb831f8515de3ec50c8ee8f015014dafd WatchSource:0}: Error finding container d20f813482abd24cd54f20e426dcfdbbb831f8515de3ec50c8ee8f015014dafd: Status 404 returned error can't find the container with id d20f813482abd24cd54f20e426dcfdbbb831f8515de3ec50c8ee8f015014dafd Sep 15 12:26:49.660242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:49.660168 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" event={"ID":"3cc89b7f-ef8a-496d-9d6f-296856a2a966","Type":"ContainerStarted","Data":"7cd18be31f397f185811a2f1b7db70ac03d96691c347487bfb900b20a38cc4f3"} Sep 15 12:26:49.662038 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:49.661980 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf4f6947-ptp6q" event={"ID":"59097b15-f7a9-457d-81a1-2de081788281","Type":"ContainerStarted","Data":"d20f813482abd24cd54f20e426dcfdbbb831f8515de3ec50c8ee8f015014dafd"} Sep 15 12:26:49.664902 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:49.664817 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wwn96" event={"ID":"e0f7776b-cd11-4862-9b0c-e638e096aaa5","Type":"ContainerStarted","Data":"5f1bce7d02c818811e792a1729c9c698fc948365f3039253ae776a640a2ab43c"} Sep 15 12:26:49.667187 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:49.667155 2550 generic.go:356] "Generic (PLEG): container finished" podID="7ca9ce81-ca99-41f1-b380-f621b12b6604" containerID="ea3a1084ebef383ab296be43464e2984c8a9325be840480376a38b3d0f5ab703" exitCode=0 Sep 15 12:26:49.667979 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:49.667955 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmwfb" event={"ID":"7ca9ce81-ca99-41f1-b380-f621b12b6604","Type":"ContainerDied","Data":"ea3a1084ebef383ab296be43464e2984c8a9325be840480376a38b3d0f5ab703"} Sep 15 12:26:50.431373 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:50.431327 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bd8db4dc9-m8kbs"] Sep 15 12:26:50.676598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:50.676331 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmwfb" event={"ID":"7ca9ce81-ca99-41f1-b380-f621b12b6604","Type":"ContainerStarted","Data":"127a021c619198b22c20bcd53b76cba620726c0e8364573923fc80525d00eca2"} Sep 15 12:26:50.676598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:50.676451 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmwfb" event={"ID":"7ca9ce81-ca99-41f1-b380-f621b12b6604","Type":"ContainerStarted","Data":"7f63626a3fc26d488d19ee33685e05317744ac134189119940c5691da4cb901e"} Sep 15 12:26:50.679264 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:50.679061 2550 generic.go:356] "Generic (PLEG): container finished" podID="7d96fad9-6b46-462f-81f9-4729c93ed38c" containerID="9c566756938c0e3cc03936fa6cf08ba96df29d384193e59d4d494ffbe1198bc3" exitCode=0 Sep 15 12:26:50.679264 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:50.679178 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerDied","Data":"9c566756938c0e3cc03936fa6cf08ba96df29d384193e59d4d494ffbe1198bc3"} Sep 15 12:26:50.681829 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:50.681511 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf4f6947-ptp6q" event={"ID":"59097b15-f7a9-457d-81a1-2de081788281","Type":"ContainerStarted","Data":"dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2"} Sep 15 12:26:50.698183 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:50.696541 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bmwfb" podStartSLOduration=6.710444202 podStartE2EDuration="8.696516369s" totalImagesPullingTime="1.883545002s" totalInitContainerRuntime="102.527165ms" isStatefulPod=true podCreationTimestamp="2026-09-15 12:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:50.695736948 +0000 UTC m=+92.050583273" watchObservedRunningTime="2026-09-15 12:26:50.696516369 +0000 UTC m=+92.051362694" Sep 15 12:26:51.690771 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:51.690653 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" event={"ID":"3cc89b7f-ef8a-496d-9d6f-296856a2a966","Type":"ContainerStarted","Data":"fd734be4f73294f6a36930753c07b828667ec6be0a0e3e7c04dc5f94dba716cf"} Sep 15 12:26:51.691307 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:51.690964 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:51.698319 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:51.698277 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" Sep 15 12:26:51.706014 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:51.705934 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79bf4f6947-ptp6q" podStartSLOduration=3.705914247 podStartE2EDuration="3.705914247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:50.738856203 +0000 UTC m=+92.093702526" watchObservedRunningTime="2026-09-15 12:26:51.705914247 +0000 UTC m=+93.060760574" Sep 15 12:26:51.706278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:51.706235 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-4hqf9" podStartSLOduration=2.57000095 podStartE2EDuration="4.706225519s" totalImagesPullingTime="2.136224569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:51.704960543 +0000 UTC m=+93.059806868" watchObservedRunningTime="2026-09-15 12:26:51.706225519 +0000 UTC m=+93.061071842" Sep 15 12:26:53.705413 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:53.705306 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerStarted","Data":"45dab78e405396bf4860b51782be74388cc7b717778ca9b314f444dd2ea19ad9"} Sep 15 12:26:53.705910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:53.705427 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerStarted","Data":"18f8056991f542e581aec4cf95842facb1a3f5d47afde42f0e8f3148045f77c3"} Sep 15 12:26:53.705910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:53.705470 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerStarted","Data":"85488b35648ccc52bcd863ae2ad334a8df8d035a2f0665e889138f26885613b8"} Sep 15 12:26:53.705910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:53.705518 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerStarted","Data":"3cb745a891ed6e9a7588d67e4ac3af1e07ed0555fae704674ffe4efb7f4a64f0"} Sep 15 12:26:54.720016 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:54.719928 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wwn96" event={"ID":"e0f7776b-cd11-4862-9b0c-e638e096aaa5","Type":"ContainerStarted","Data":"60e0657f26879746e525127a719157039118a0751f7214ce8419603f92179dca"} Sep 15 12:26:54.724023 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:54.723989 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerStarted","Data":"6e898aa67f9589a70b6458cd466b2ed0f4f02ed4cd52bbdb0550893f3e7b5718"} Sep 15 12:26:54.738688 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:54.738617 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wwn96" podStartSLOduration=17.384883417 podStartE2EDuration="23.738597105s" totalImagesPullingTime="6.353713688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:54.736685755 +0000 UTC m=+96.091532079" watchObservedRunningTime="2026-09-15 12:26:54.738597105 +0000 UTC m=+96.093443429" Sep 15 12:26:55.733327 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:55.733241 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d96fad9-6b46-462f-81f9-4729c93ed38c","Type":"ContainerStarted","Data":"a870be1ee3b0359d40bdf2c87d9605088fd5ed5f8a2d7122f9b980e90f2bd6d0"} Sep 15 12:26:55.760197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:55.760127 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.219026435 podStartE2EDuration="12.76008813s" totalImagesPullingTime="5.500753329s" totalInitContainerRuntime="40.308366ms" isStatefulPod=false podCreationTimestamp="2026-09-15 12:26:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:26:55.758703753 +0000 UTC m=+97.113550075" watchObservedRunningTime="2026-09-15 12:26:55.76008813 +0000 UTC m=+97.114934453" Sep 15 12:26:57.502284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:57.502251 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dr79g" Sep 15 12:26:59.039549 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:59.039502 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:59.039549 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:59.039558 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:59.044115 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:59.044091 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:59.752692 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:59.752654 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:26:59.800701 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:26:59.800663 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c7f465b9d-skxb6"] Sep 15 12:27:08.786085 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:08.786045 2550 generic.go:356] "Generic (PLEG): container finished" podID="c60e7f81-7308-463e-a85d-b46103f4fa9a" containerID="039f9a5442c8aec9f5913145fa54c0a96b9b121b415989aaddb768fa797b4ccd" exitCode=0 Sep 15 12:27:08.786590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:08.786119 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" event={"ID":"c60e7f81-7308-463e-a85d-b46103f4fa9a","Type":"ContainerDied","Data":"039f9a5442c8aec9f5913145fa54c0a96b9b121b415989aaddb768fa797b4ccd"} Sep 15 12:27:08.786590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:08.786497 2550 scope.go:117] "RemoveContainer" containerID="039f9a5442c8aec9f5913145fa54c0a96b9b121b415989aaddb768fa797b4ccd" Sep 15 12:27:09.792272 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:09.792234 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-84pvr" event={"ID":"c60e7f81-7308-463e-a85d-b46103f4fa9a","Type":"ContainerStarted","Data":"325a40169d2631e67a1c99ac053e9035102fcf8aca70c30691f2cdcb26fcba83"} Sep 15 12:27:15.454283 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.454209 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" podUID="80321730-1357-442e-9eb9-c72ce4cbf85b" containerName="registry" containerID="cri-o://e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43" gracePeriod=30 Sep 15 12:27:15.738737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.738652 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:27:15.768042 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768010 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn4bv\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-kube-api-access-nn4bv\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768188 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768116 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80321730-1357-442e-9eb9-c72ce4cbf85b-ca-trust-extracted\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768194 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-image-registry-private-configuration\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768221 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-certificates\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768258 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768286 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-trusted-ca\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768440 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768315 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-installation-pull-secrets\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768440 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768358 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-bound-sa-token\") pod \"80321730-1357-442e-9eb9-c72ce4cbf85b\" (UID: \"80321730-1357-442e-9eb9-c72ce4cbf85b\") " Sep 15 12:27:15.768910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768836 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:27:15.769029 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.768949 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:27:15.770945 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.770905 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:27:15.771047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.770985 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:27:15.771112 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.771090 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:27:15.771168 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.771131 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-kube-api-access-nn4bv" (OuterVolumeSpecName: "kube-api-access-nn4bv") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "kube-api-access-nn4bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:27:15.771643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.771612 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:27:15.777181 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.777155 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80321730-1357-442e-9eb9-c72ce4cbf85b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "80321730-1357-442e-9eb9-c72ce4cbf85b" (UID: "80321730-1357-442e-9eb9-c72ce4cbf85b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:27:15.817415 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.817362 2550 generic.go:356] "Generic (PLEG): container finished" podID="80321730-1357-442e-9eb9-c72ce4cbf85b" containerID="e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43" exitCode=0 Sep 15 12:27:15.817572 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.817423 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" Sep 15 12:27:15.817572 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.817447 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" event={"ID":"80321730-1357-442e-9eb9-c72ce4cbf85b","Type":"ContainerDied","Data":"e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43"} Sep 15 12:27:15.817572 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.817493 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bd8db4dc9-m8kbs" event={"ID":"80321730-1357-442e-9eb9-c72ce4cbf85b","Type":"ContainerDied","Data":"9e182118d26fa2b08e714d543a96649d632a5aecf1a264d2c2d47ec7cc4d888a"} Sep 15 12:27:15.817572 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.817514 2550 scope.go:117] "RemoveContainer" containerID="e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43" Sep 15 12:27:15.826195 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.826177 2550 scope.go:117] "RemoveContainer" containerID="e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43" Sep 15 12:27:15.826526 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:27:15.826503 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43\": container with ID starting with e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43 not found: ID does not exist" containerID="e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43" Sep 15 12:27:15.826602 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.826532 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43"} err="failed to get container status \"e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43\": rpc error: code = NotFound desc = could not find container \"e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43\": container with ID starting with e3f545dab3364978ad088ae9e5b65f6c1fd1f5de6feaab4c1dc39918a3e4ca43 not found: ID does not exist" Sep 15 12:27:15.837100 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.837072 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bd8db4dc9-m8kbs"] Sep 15 12:27:15.840433 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.840398 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bd8db4dc9-m8kbs"] Sep 15 12:27:15.869432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869396 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-trusted-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:15.869432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869424 2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-installation-pull-secrets\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:15.869432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869433 2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-bound-sa-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:15.869616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869443 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn4bv\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-kube-api-access-nn4bv\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:15.869616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869451 2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80321730-1357-442e-9eb9-c72ce4cbf85b-ca-trust-extracted\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:15.869616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869460 2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80321730-1357-442e-9eb9-c72ce4cbf85b-image-registry-private-configuration\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:15.869616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869469 2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-certificates\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:15.869616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:15.869478 2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80321730-1357-442e-9eb9-c72ce4cbf85b-registry-tls\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:17.153371 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:17.153334 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80321730-1357-442e-9eb9-c72ce4cbf85b" path="/var/lib/kubelet/pods/80321730-1357-442e-9eb9-c72ce4cbf85b/volumes" Sep 15 12:27:18.832425 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:18.832361 2550 generic.go:356] "Generic (PLEG): container finished" podID="18be3ccf-b37c-401c-a169-cc2820e6103f" containerID="293592c90c23192d8011e250fdbe3d2d496b3a29557c00ff936c2edf17a89cdf" exitCode=0 Sep 15 12:27:18.832793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:18.832441 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" event={"ID":"18be3ccf-b37c-401c-a169-cc2820e6103f","Type":"ContainerDied","Data":"293592c90c23192d8011e250fdbe3d2d496b3a29557c00ff936c2edf17a89cdf"} Sep 15 12:27:18.832851 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:18.832805 2550 scope.go:117] "RemoveContainer" containerID="293592c90c23192d8011e250fdbe3d2d496b3a29557c00ff936c2edf17a89cdf" Sep 15 12:27:19.838809 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:19.838775 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n5dv8" event={"ID":"18be3ccf-b37c-401c-a169-cc2820e6103f","Type":"ContainerStarted","Data":"0f7a4e317fa04c45ffc6fedc9bd60cbc025cb8e4fbd5d7ea752c35f8d43fd6d2"} Sep 15 12:27:24.820817 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:24.820777 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c7f465b9d-skxb6" podUID="7f9d15ec-ec26-46bd-9605-d0739e2c5f34" containerName="console" containerID="cri-o://dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82" gracePeriod=15 Sep 15 12:27:25.098935 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.098866 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:27:25.147814 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.147771 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x29rj\" (UniqueName: \"kubernetes.io/projected/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-kube-api-access-x29rj\") pod \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " Sep 15 12:27:25.147814 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.147818 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-service-ca\") pod \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " Sep 15 12:27:25.148058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.147876 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-oauth-config\") pod \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " Sep 15 12:27:25.148058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.147916 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-config\") pod \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " Sep 15 12:27:25.148058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.147952 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-oauth-serving-cert\") pod \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " Sep 15 12:27:25.148058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.147982 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-trusted-ca-bundle\") pod \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " Sep 15 12:27:25.148058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.148013 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-serving-cert\") pod \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\" (UID: \"7f9d15ec-ec26-46bd-9605-d0739e2c5f34\") " Sep 15 12:27:25.148305 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.148260 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-service-ca" (OuterVolumeSpecName: "service-ca") pod "7f9d15ec-ec26-46bd-9605-d0739e2c5f34" (UID: "7f9d15ec-ec26-46bd-9605-d0739e2c5f34"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:27:25.148440 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.148413 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-config" (OuterVolumeSpecName: "console-config") pod "7f9d15ec-ec26-46bd-9605-d0739e2c5f34" (UID: "7f9d15ec-ec26-46bd-9605-d0739e2c5f34"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:27:25.148612 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.148574 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7f9d15ec-ec26-46bd-9605-d0739e2c5f34" (UID: "7f9d15ec-ec26-46bd-9605-d0739e2c5f34"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:27:25.148612 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.148589 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7f9d15ec-ec26-46bd-9605-d0739e2c5f34" (UID: "7f9d15ec-ec26-46bd-9605-d0739e2c5f34"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:27:25.150303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.150278 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7f9d15ec-ec26-46bd-9605-d0739e2c5f34" (UID: "7f9d15ec-ec26-46bd-9605-d0739e2c5f34"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:27:25.150536 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.150513 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7f9d15ec-ec26-46bd-9605-d0739e2c5f34" (UID: "7f9d15ec-ec26-46bd-9605-d0739e2c5f34"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:27:25.150610 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.150552 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-kube-api-access-x29rj" (OuterVolumeSpecName: "kube-api-access-x29rj") pod "7f9d15ec-ec26-46bd-9605-d0739e2c5f34" (UID: "7f9d15ec-ec26-46bd-9605-d0739e2c5f34"). InnerVolumeSpecName "kube-api-access-x29rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:27:25.249666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.249629 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-oauth-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:25.249666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.249660 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-trusted-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:25.249666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.249670 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:25.249911 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.249680 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29rj\" (UniqueName: \"kubernetes.io/projected/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-kube-api-access-x29rj\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:25.249911 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.249690 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:25.249911 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.249701 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-oauth-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:25.249911 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.249708 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f9d15ec-ec26-46bd-9605-d0739e2c5f34-console-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:27:25.864136 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.864099 2550 generic.go:356] "Generic (PLEG): container finished" podID="7f9d15ec-ec26-46bd-9605-d0739e2c5f34" containerID="dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82" exitCode=0 Sep 15 12:27:25.864643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.864156 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c7f465b9d-skxb6" Sep 15 12:27:25.864643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.864166 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7f465b9d-skxb6" event={"ID":"7f9d15ec-ec26-46bd-9605-d0739e2c5f34","Type":"ContainerDied","Data":"dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82"} Sep 15 12:27:25.864643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.864205 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c7f465b9d-skxb6" event={"ID":"7f9d15ec-ec26-46bd-9605-d0739e2c5f34","Type":"ContainerDied","Data":"72a3eaa5e56752bdadf25aa400a6e969ff6a58fa9f54cb5e669658dd63f3c6da"} Sep 15 12:27:25.864643 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.864225 2550 scope.go:117] "RemoveContainer" containerID="dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82" Sep 15 12:27:25.872232 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.872213 2550 scope.go:117] "RemoveContainer" containerID="dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82" Sep 15 12:27:25.872572 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:27:25.872550 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82\": container with ID starting with dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82 not found: ID does not exist" containerID="dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82" Sep 15 12:27:25.872639 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.872582 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82"} err="failed to get container status \"dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82\": rpc error: code = NotFound desc = could not find container \"dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82\": container with ID starting with dead0e6bb35bdc627423cf2cdd43ae24407a877677ba66a76bf0491365433d82 not found: ID does not exist" Sep 15 12:27:25.878916 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.878885 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c7f465b9d-skxb6"] Sep 15 12:27:25.882435 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:25.882410 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c7f465b9d-skxb6"] Sep 15 12:27:27.152081 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:27:27.152041 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f9d15ec-ec26-46bd-9605-d0739e2c5f34" path="/var/lib/kubelet/pods/7f9d15ec-ec26-46bd-9605-d0739e2c5f34/volumes" Sep 15 12:28:07.066598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.066558 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd"] Sep 15 12:28:07.067193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.066857 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9d15ec-ec26-46bd-9605-d0739e2c5f34" containerName="console" Sep 15 12:28:07.067193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.066869 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9d15ec-ec26-46bd-9605-d0739e2c5f34" containerName="console" Sep 15 12:28:07.067193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.066883 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80321730-1357-442e-9eb9-c72ce4cbf85b" containerName="registry" Sep 15 12:28:07.067193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.066889 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="80321730-1357-442e-9eb9-c72ce4cbf85b" containerName="registry" Sep 15 12:28:07.067193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.066945 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9d15ec-ec26-46bd-9605-d0739e2c5f34" containerName="console" Sep 15 12:28:07.067193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.066953 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="80321730-1357-442e-9eb9-c72ce4cbf85b" containerName="registry" Sep 15 12:28:07.071768 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.071739 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.074331 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.074307 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 15 12:28:07.074480 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.074330 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 15 12:28:07.074480 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.074471 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 15 12:28:07.074600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.074490 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 15 12:28:07.074686 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.074662 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zf5v4\"" Sep 15 12:28:07.074790 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.074741 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 15 12:28:07.080655 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.080023 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 15 12:28:07.082872 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.082843 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd"] Sep 15 12:28:07.092406 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092363 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.092601 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092429 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-secret-telemeter-client\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.092601 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092468 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-serving-certs-ca-bundle\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.092719 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092593 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-federate-client-tls\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.092719 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092635 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-metrics-client-ca\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.092719 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092664 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.092719 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092697 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-telemeter-client-tls\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.092848 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.092772 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhmsg\" (UniqueName: \"kubernetes.io/projected/74925c27-0ed3-402a-b309-9d1a57664f8d-kube-api-access-vhmsg\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.193598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.193552 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-serving-certs-ca-bundle\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.193800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.193620 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-federate-client-tls\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.193800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.193637 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-metrics-client-ca\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.193800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.193655 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.193800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.193676 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-telemeter-client-tls\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.193800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.193702 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhmsg\" (UniqueName: \"kubernetes.io/projected/74925c27-0ed3-402a-b309-9d1a57664f8d-kube-api-access-vhmsg\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.193800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.193769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.194696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.194242 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-secret-telemeter-client\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.194978 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.194953 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-serving-certs-ca-bundle\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.200663 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.195492 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-metrics-client-ca\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.200663 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.196279 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74925c27-0ed3-402a-b309-9d1a57664f8d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.200663 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.197199 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-federate-client-tls\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.200663 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.197201 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-telemeter-client-tls\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.200663 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.198443 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.203963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.203936 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/74925c27-0ed3-402a-b309-9d1a57664f8d-secret-telemeter-client\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.205938 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.205908 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhmsg\" (UniqueName: \"kubernetes.io/projected/74925c27-0ed3-402a-b309-9d1a57664f8d-kube-api-access-vhmsg\") pod \"telemeter-client-5d989b5bdd-zm4fd\" (UID: \"74925c27-0ed3-402a-b309-9d1a57664f8d\") " pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.383810 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.383710 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" Sep 15 12:28:07.664972 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:07.664867 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd"] Sep 15 12:28:07.667644 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:28:07.667616 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74925c27_0ed3_402a_b309_9d1a57664f8d.slice/crio-d2c73de738b6aa30094704f3f4b3fcba84350d1141d779871352ac8ca3fcda13 WatchSource:0}: Error finding container d2c73de738b6aa30094704f3f4b3fcba84350d1141d779871352ac8ca3fcda13: Status 404 returned error can't find the container with id d2c73de738b6aa30094704f3f4b3fcba84350d1141d779871352ac8ca3fcda13 Sep 15 12:28:08.027902 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:08.027870 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" event={"ID":"74925c27-0ed3-402a-b309-9d1a57664f8d","Type":"ContainerStarted","Data":"d2c73de738b6aa30094704f3f4b3fcba84350d1141d779871352ac8ca3fcda13"} Sep 15 12:28:10.038375 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.038338 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" event={"ID":"74925c27-0ed3-402a-b309-9d1a57664f8d","Type":"ContainerStarted","Data":"d44e7f7e54e3eb7967505e6ac271d8a665c6341707a5b4db213546885fc27d35"} Sep 15 12:28:10.038910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.038422 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" event={"ID":"74925c27-0ed3-402a-b309-9d1a57664f8d","Type":"ContainerStarted","Data":"7202d1fabd066ca16eeb1c98102b7d9d55ab4ddee26fb57131847dd4864ee42b"} Sep 15 12:28:10.038910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.038444 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" event={"ID":"74925c27-0ed3-402a-b309-9d1a57664f8d","Type":"ContainerStarted","Data":"587e0becad63b6710dca39ab8141f398a88de72788d32b10b23dbb4f42c718d4"} Sep 15 12:28:10.058456 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.058354 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d989b5bdd-zm4fd" podStartSLOduration=1.379975873 podStartE2EDuration="3.058302516s" totalImagesPullingTime="1.678326643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:28:10.058272507 +0000 UTC m=+171.413118831" watchObservedRunningTime="2026-09-15 12:28:10.058302516 +0000 UTC m=+171.413148839" Sep 15 12:28:10.140973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.139027 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f6679d87-srvxj"] Sep 15 12:28:10.142662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.142634 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.152003 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.151971 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f6679d87-srvxj"] Sep 15 12:28:10.316536 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.316436 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-oauth-config\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.316536 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.316480 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-trusted-ca-bundle\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.316759 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.316556 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-serving-cert\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.316759 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.316618 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-oauth-serving-cert\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.316759 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.316661 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-console-config\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.316759 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.316692 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-service-ca\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.316759 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.316721 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7j45\" (UniqueName: \"kubernetes.io/projected/5b599a48-85ef-4e21-b32d-e74d56595497-kube-api-access-b7j45\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.417276 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.417243 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7j45\" (UniqueName: \"kubernetes.io/projected/5b599a48-85ef-4e21-b32d-e74d56595497-kube-api-access-b7j45\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.417276 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.417292 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-oauth-config\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.417542 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.417316 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-trusted-ca-bundle\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.417542 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.417364 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-serving-cert\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.417542 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.417423 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-oauth-serving-cert\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.417542 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.417527 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-console-config\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.417817 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.417570 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-service-ca\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.418255 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.418232 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-service-ca\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.418355 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.418295 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-console-config\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.418355 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.418325 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-oauth-serving-cert\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.418476 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.418422 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-trusted-ca-bundle\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.419965 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.419940 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-oauth-config\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.420057 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.420018 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-serving-cert\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.425831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.425810 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7j45\" (UniqueName: \"kubernetes.io/projected/5b599a48-85ef-4e21-b32d-e74d56595497-kube-api-access-b7j45\") pod \"console-67f6679d87-srvxj\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.453145 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.453101 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:10.724925 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:10.724855 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f6679d87-srvxj"] Sep 15 12:28:10.728510 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:28:10.728483 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b599a48_85ef_4e21_b32d_e74d56595497.slice/crio-43055e43648578417b7016193be6cfdd3c6abdb995b3d1b7f77079d9f751e1a1 WatchSource:0}: Error finding container 43055e43648578417b7016193be6cfdd3c6abdb995b3d1b7f77079d9f751e1a1: Status 404 returned error can't find the container with id 43055e43648578417b7016193be6cfdd3c6abdb995b3d1b7f77079d9f751e1a1 Sep 15 12:28:11.044172 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:11.044133 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f6679d87-srvxj" event={"ID":"5b599a48-85ef-4e21-b32d-e74d56595497","Type":"ContainerStarted","Data":"5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14"} Sep 15 12:28:11.044676 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:11.044190 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f6679d87-srvxj" event={"ID":"5b599a48-85ef-4e21-b32d-e74d56595497","Type":"ContainerStarted","Data":"43055e43648578417b7016193be6cfdd3c6abdb995b3d1b7f77079d9f751e1a1"} Sep 15 12:28:11.061147 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:11.061092 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f6679d87-srvxj" podStartSLOduration=1.061073685 podStartE2EDuration="1.061073685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:28:11.059922775 +0000 UTC m=+172.414769097" watchObservedRunningTime="2026-09-15 12:28:11.061073685 +0000 UTC m=+172.415920008" Sep 15 12:28:20.453776 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:20.453707 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:20.454197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:20.453797 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:20.458583 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:20.458549 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:21.087404 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.087353 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:21.123808 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.123772 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bf4f6947-ptp6q"] Sep 15 12:28:21.181656 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.181621 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66cbd5594-5jsn9"] Sep 15 12:28:21.188745 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.188714 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.193603 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.193567 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cbd5594-5jsn9"] Sep 15 12:28:21.202846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.202810 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-oauth-config\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.202968 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.202859 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-service-ca\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.202968 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.202960 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgc5\" (UniqueName: \"kubernetes.io/projected/850d66e7-4177-47b9-a91d-3ab064adb273-kube-api-access-hjgc5\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.203058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.202989 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-console-config\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.203101 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.203063 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-oauth-serving-cert\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.203143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.203100 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-serving-cert\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.203143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.203123 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-trusted-ca-bundle\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.304162 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.304122 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-oauth-serving-cert\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.304162 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.304175 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-serving-cert\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.304451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.304198 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-trusted-ca-bundle\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.304451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.304257 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-oauth-config\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.304451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.304281 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-service-ca\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.304451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.304342 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgc5\" (UniqueName: \"kubernetes.io/projected/850d66e7-4177-47b9-a91d-3ab064adb273-kube-api-access-hjgc5\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.304451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.304369 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-console-config\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.305199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.305168 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-service-ca\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.305304 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.305182 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-oauth-serving-cert\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.305372 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.305338 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-trusted-ca-bundle\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.305372 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.305359 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-console-config\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.306781 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.306757 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-oauth-config\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.306905 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.306884 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-serving-cert\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.311800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.311777 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgc5\" (UniqueName: \"kubernetes.io/projected/850d66e7-4177-47b9-a91d-3ab064adb273-kube-api-access-hjgc5\") pod \"console-66cbd5594-5jsn9\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.501065 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.501022 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:21.774334 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:21.774256 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66cbd5594-5jsn9"] Sep 15 12:28:21.777258 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:28:21.777216 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850d66e7_4177_47b9_a91d_3ab064adb273.slice/crio-bff7248850ec6947084ed3681f7fc995cefc777b36c1e1edbc325dfb7917369f WatchSource:0}: Error finding container bff7248850ec6947084ed3681f7fc995cefc777b36c1e1edbc325dfb7917369f: Status 404 returned error can't find the container with id bff7248850ec6947084ed3681f7fc995cefc777b36c1e1edbc325dfb7917369f Sep 15 12:28:22.090219 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:22.090121 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cbd5594-5jsn9" event={"ID":"850d66e7-4177-47b9-a91d-3ab064adb273","Type":"ContainerStarted","Data":"d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d"} Sep 15 12:28:22.090219 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:22.090180 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cbd5594-5jsn9" event={"ID":"850d66e7-4177-47b9-a91d-3ab064adb273","Type":"ContainerStarted","Data":"bff7248850ec6947084ed3681f7fc995cefc777b36c1e1edbc325dfb7917369f"} Sep 15 12:28:22.105937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:22.105888 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66cbd5594-5jsn9" podStartSLOduration=1.105873305 podStartE2EDuration="1.105873305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:28:22.104571559 +0000 UTC m=+183.459417881" watchObservedRunningTime="2026-09-15 12:28:22.105873305 +0000 UTC m=+183.460719627" Sep 15 12:28:31.501551 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:31.501500 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:31.502031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:31.501584 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:31.506497 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:31.506470 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:32.140394 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:32.140336 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:28:32.185687 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:32.185652 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f6679d87-srvxj"] Sep 15 12:28:46.145926 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.145868 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79bf4f6947-ptp6q" podUID="59097b15-f7a9-457d-81a1-2de081788281" containerName="console" containerID="cri-o://dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2" gracePeriod=15 Sep 15 12:28:46.396466 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.396400 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:28:46.519507 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.519467 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-oauth-config\") pod \"59097b15-f7a9-457d-81a1-2de081788281\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " Sep 15 12:28:46.519704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.519528 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-trusted-ca-bundle\") pod \"59097b15-f7a9-457d-81a1-2de081788281\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " Sep 15 12:28:46.519704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.519587 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-console-config\") pod \"59097b15-f7a9-457d-81a1-2de081788281\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " Sep 15 12:28:46.519704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.519613 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-oauth-serving-cert\") pod \"59097b15-f7a9-457d-81a1-2de081788281\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " Sep 15 12:28:46.519704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.519637 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9gq6\" (UniqueName: \"kubernetes.io/projected/59097b15-f7a9-457d-81a1-2de081788281-kube-api-access-x9gq6\") pod \"59097b15-f7a9-457d-81a1-2de081788281\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " Sep 15 12:28:46.519704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.519661 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-service-ca\") pod \"59097b15-f7a9-457d-81a1-2de081788281\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " Sep 15 12:28:46.519704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.519687 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-serving-cert\") pod \"59097b15-f7a9-457d-81a1-2de081788281\" (UID: \"59097b15-f7a9-457d-81a1-2de081788281\") " Sep 15 12:28:46.520059 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.520029 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-console-config" (OuterVolumeSpecName: "console-config") pod "59097b15-f7a9-457d-81a1-2de081788281" (UID: "59097b15-f7a9-457d-81a1-2de081788281"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:46.520126 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.520052 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "59097b15-f7a9-457d-81a1-2de081788281" (UID: "59097b15-f7a9-457d-81a1-2de081788281"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:46.520126 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.520047 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "59097b15-f7a9-457d-81a1-2de081788281" (UID: "59097b15-f7a9-457d-81a1-2de081788281"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:46.520126 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.520103 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-service-ca" (OuterVolumeSpecName: "service-ca") pod "59097b15-f7a9-457d-81a1-2de081788281" (UID: "59097b15-f7a9-457d-81a1-2de081788281"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:46.521822 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.521790 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "59097b15-f7a9-457d-81a1-2de081788281" (UID: "59097b15-f7a9-457d-81a1-2de081788281"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:28:46.521907 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.521792 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59097b15-f7a9-457d-81a1-2de081788281-kube-api-access-x9gq6" (OuterVolumeSpecName: "kube-api-access-x9gq6") pod "59097b15-f7a9-457d-81a1-2de081788281" (UID: "59097b15-f7a9-457d-81a1-2de081788281"). InnerVolumeSpecName "kube-api-access-x9gq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:28:46.521907 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.521859 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "59097b15-f7a9-457d-81a1-2de081788281" (UID: "59097b15-f7a9-457d-81a1-2de081788281"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:28:46.620473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.620435 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-console-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:46.620473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.620466 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-oauth-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:46.620473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.620475 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9gq6\" (UniqueName: \"kubernetes.io/projected/59097b15-f7a9-457d-81a1-2de081788281-kube-api-access-x9gq6\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:46.620473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.620486 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:46.620739 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.620496 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:46.620739 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.620504 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/59097b15-f7a9-457d-81a1-2de081788281-console-oauth-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:46.620739 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:46.620512 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59097b15-f7a9-457d-81a1-2de081788281-trusted-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:47.192715 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.192677 2550 generic.go:356] "Generic (PLEG): container finished" podID="59097b15-f7a9-457d-81a1-2de081788281" containerID="dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2" exitCode=0 Sep 15 12:28:47.193163 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.192724 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bf4f6947-ptp6q" Sep 15 12:28:47.193163 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.192735 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf4f6947-ptp6q" event={"ID":"59097b15-f7a9-457d-81a1-2de081788281","Type":"ContainerDied","Data":"dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2"} Sep 15 12:28:47.193163 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.192785 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bf4f6947-ptp6q" event={"ID":"59097b15-f7a9-457d-81a1-2de081788281","Type":"ContainerDied","Data":"d20f813482abd24cd54f20e426dcfdbbb831f8515de3ec50c8ee8f015014dafd"} Sep 15 12:28:47.193163 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.192812 2550 scope.go:117] "RemoveContainer" containerID="dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2" Sep 15 12:28:47.201062 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.201042 2550 scope.go:117] "RemoveContainer" containerID="dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2" Sep 15 12:28:47.201420 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:28:47.201394 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2\": container with ID starting with dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2 not found: ID does not exist" containerID="dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2" Sep 15 12:28:47.201486 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.201438 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2"} err="failed to get container status \"dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2\": rpc error: code = NotFound desc = could not find container \"dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2\": container with ID starting with dd4455b03d3227565c24c936975b9cba99ee8c97a8e8bd24b6b631712f134fe2 not found: ID does not exist" Sep 15 12:28:47.207624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.207598 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bf4f6947-ptp6q"] Sep 15 12:28:47.209662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:47.209637 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79bf4f6947-ptp6q"] Sep 15 12:28:49.152909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:49.152874 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59097b15-f7a9-457d-81a1-2de081788281" path="/var/lib/kubelet/pods/59097b15-f7a9-457d-81a1-2de081788281/volumes" Sep 15 12:28:57.211227 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.211177 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67f6679d87-srvxj" podUID="5b599a48-85ef-4e21-b32d-e74d56595497" containerName="console" containerID="cri-o://5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14" gracePeriod=15 Sep 15 12:28:57.454244 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.454209 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:57.623900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.623865 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-serving-cert\") pod \"5b599a48-85ef-4e21-b32d-e74d56595497\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " Sep 15 12:28:57.624099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.623920 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-console-config\") pod \"5b599a48-85ef-4e21-b32d-e74d56595497\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " Sep 15 12:28:57.624099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.623945 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7j45\" (UniqueName: \"kubernetes.io/projected/5b599a48-85ef-4e21-b32d-e74d56595497-kube-api-access-b7j45\") pod \"5b599a48-85ef-4e21-b32d-e74d56595497\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " Sep 15 12:28:57.624099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.623977 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-trusted-ca-bundle\") pod \"5b599a48-85ef-4e21-b32d-e74d56595497\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " Sep 15 12:28:57.624099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.624013 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-oauth-serving-cert\") pod \"5b599a48-85ef-4e21-b32d-e74d56595497\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " Sep 15 12:28:57.624099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.624039 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-service-ca\") pod \"5b599a48-85ef-4e21-b32d-e74d56595497\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " Sep 15 12:28:57.624099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.624067 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-oauth-config\") pod \"5b599a48-85ef-4e21-b32d-e74d56595497\" (UID: \"5b599a48-85ef-4e21-b32d-e74d56595497\") " Sep 15 12:28:57.624594 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.624544 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-console-config" (OuterVolumeSpecName: "console-config") pod "5b599a48-85ef-4e21-b32d-e74d56595497" (UID: "5b599a48-85ef-4e21-b32d-e74d56595497"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:57.624594 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.624576 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-service-ca" (OuterVolumeSpecName: "service-ca") pod "5b599a48-85ef-4e21-b32d-e74d56595497" (UID: "5b599a48-85ef-4e21-b32d-e74d56595497"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:57.624594 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.624589 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5b599a48-85ef-4e21-b32d-e74d56595497" (UID: "5b599a48-85ef-4e21-b32d-e74d56595497"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:57.624794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.624626 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5b599a48-85ef-4e21-b32d-e74d56595497" (UID: "5b599a48-85ef-4e21-b32d-e74d56595497"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:28:57.626232 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.626203 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b599a48-85ef-4e21-b32d-e74d56595497-kube-api-access-b7j45" (OuterVolumeSpecName: "kube-api-access-b7j45") pod "5b599a48-85ef-4e21-b32d-e74d56595497" (UID: "5b599a48-85ef-4e21-b32d-e74d56595497"). InnerVolumeSpecName "kube-api-access-b7j45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:28:57.626351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.626237 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5b599a48-85ef-4e21-b32d-e74d56595497" (UID: "5b599a48-85ef-4e21-b32d-e74d56595497"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:28:57.626351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.626308 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5b599a48-85ef-4e21-b32d-e74d56595497" (UID: "5b599a48-85ef-4e21-b32d-e74d56595497"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:28:57.725106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.725062 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7j45\" (UniqueName: \"kubernetes.io/projected/5b599a48-85ef-4e21-b32d-e74d56595497-kube-api-access-b7j45\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:57.725106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.725098 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-trusted-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:57.725106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.725112 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-oauth-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:57.725450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.725125 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:57.725450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.725136 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-oauth-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:57.725450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.725147 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b599a48-85ef-4e21-b32d-e74d56595497-console-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:57.725450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:57.725160 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5b599a48-85ef-4e21-b32d-e74d56595497-console-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:28:58.239332 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.239293 2550 generic.go:356] "Generic (PLEG): container finished" podID="5b599a48-85ef-4e21-b32d-e74d56595497" containerID="5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14" exitCode=0 Sep 15 12:28:58.239735 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.239339 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f6679d87-srvxj" Sep 15 12:28:58.239735 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.239390 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f6679d87-srvxj" event={"ID":"5b599a48-85ef-4e21-b32d-e74d56595497","Type":"ContainerDied","Data":"5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14"} Sep 15 12:28:58.239735 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.239446 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f6679d87-srvxj" event={"ID":"5b599a48-85ef-4e21-b32d-e74d56595497","Type":"ContainerDied","Data":"43055e43648578417b7016193be6cfdd3c6abdb995b3d1b7f77079d9f751e1a1"} Sep 15 12:28:58.240416 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.239472 2550 scope.go:117] "RemoveContainer" containerID="5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14" Sep 15 12:28:58.249003 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.248977 2550 scope.go:117] "RemoveContainer" containerID="5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14" Sep 15 12:28:58.249261 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:28:58.249238 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14\": container with ID starting with 5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14 not found: ID does not exist" containerID="5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14" Sep 15 12:28:58.249322 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.249273 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14"} err="failed to get container status \"5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14\": rpc error: code = NotFound desc = could not find container \"5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14\": container with ID starting with 5fdac92d896e9f9c8f6cae2da22d9c70c07549e1027705ac0f04de533297db14 not found: ID does not exist" Sep 15 12:28:58.260812 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.260779 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f6679d87-srvxj"] Sep 15 12:28:58.264886 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:58.264861 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f6679d87-srvxj"] Sep 15 12:28:59.152567 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:28:59.152532 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b599a48-85ef-4e21-b32d-e74d56595497" path="/var/lib/kubelet/pods/5b599a48-85ef-4e21-b32d-e74d56595497/volumes" Sep 15 12:29:59.947793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.947756 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n"] Sep 15 12:29:59.948236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.948057 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59097b15-f7a9-457d-81a1-2de081788281" containerName="console" Sep 15 12:29:59.948236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.948067 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59097b15-f7a9-457d-81a1-2de081788281" containerName="console" Sep 15 12:29:59.948236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.948087 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b599a48-85ef-4e21-b32d-e74d56595497" containerName="console" Sep 15 12:29:59.948236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.948092 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b599a48-85ef-4e21-b32d-e74d56595497" containerName="console" Sep 15 12:29:59.948236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.948146 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59097b15-f7a9-457d-81a1-2de081788281" containerName="console" Sep 15 12:29:59.948236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.948156 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b599a48-85ef-4e21-b32d-e74d56595497" containerName="console" Sep 15 12:29:59.951015 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.950996 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:29:59.953071 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.953045 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 15 12:29:59.953204 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.953085 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-52qvz\"" Sep 15 12:29:59.953204 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.953107 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:29:59.965550 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:29:59.965520 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n"] Sep 15 12:30:00.139303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.139264 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfj9z\" (UniqueName: \"kubernetes.io/projected/fbe8ebe6-ecbc-42ed-a014-924798497316-kube-api-access-sfj9z\") pod \"cert-manager-operator-controller-manager-55c444bf86-2j55n\" (UID: \"fbe8ebe6-ecbc-42ed-a014-924798497316\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:30:00.139500 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.139329 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe8ebe6-ecbc-42ed-a014-924798497316-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2j55n\" (UID: \"fbe8ebe6-ecbc-42ed-a014-924798497316\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:30:00.240415 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.240371 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe8ebe6-ecbc-42ed-a014-924798497316-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2j55n\" (UID: \"fbe8ebe6-ecbc-42ed-a014-924798497316\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:30:00.240564 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.240461 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfj9z\" (UniqueName: \"kubernetes.io/projected/fbe8ebe6-ecbc-42ed-a014-924798497316-kube-api-access-sfj9z\") pod \"cert-manager-operator-controller-manager-55c444bf86-2j55n\" (UID: \"fbe8ebe6-ecbc-42ed-a014-924798497316\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:30:00.240781 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.240761 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbe8ebe6-ecbc-42ed-a014-924798497316-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2j55n\" (UID: \"fbe8ebe6-ecbc-42ed-a014-924798497316\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:30:00.249684 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.249660 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfj9z\" (UniqueName: \"kubernetes.io/projected/fbe8ebe6-ecbc-42ed-a014-924798497316-kube-api-access-sfj9z\") pod \"cert-manager-operator-controller-manager-55c444bf86-2j55n\" (UID: \"fbe8ebe6-ecbc-42ed-a014-924798497316\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:30:00.260557 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.260532 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" Sep 15 12:30:00.538364 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:00.538333 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n"] Sep 15 12:30:00.540517 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:30:00.540487 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe8ebe6_ecbc_42ed_a014_924798497316.slice/crio-6ffc0e657f6ce66456fec88b3e9dda9241f5b376eb7e8b3b9e67f4fe5c869796 WatchSource:0}: Error finding container 6ffc0e657f6ce66456fec88b3e9dda9241f5b376eb7e8b3b9e67f4fe5c869796: Status 404 returned error can't find the container with id 6ffc0e657f6ce66456fec88b3e9dda9241f5b376eb7e8b3b9e67f4fe5c869796 Sep 15 12:30:01.492920 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:01.492879 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" event={"ID":"fbe8ebe6-ecbc-42ed-a014-924798497316","Type":"ContainerStarted","Data":"6ffc0e657f6ce66456fec88b3e9dda9241f5b376eb7e8b3b9e67f4fe5c869796"} Sep 15 12:30:03.504242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:03.504196 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" event={"ID":"fbe8ebe6-ecbc-42ed-a014-924798497316","Type":"ContainerStarted","Data":"7b268004ba1c7dd3e6d06b586d8f5f808c3bcb7f91b33cedf2edf6b364003126"} Sep 15 12:30:03.524798 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:03.524743 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2j55n" podStartSLOduration=2.105356358 podStartE2EDuration="4.524725724s" totalImagesPullingTime="2.419369366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:03.523281133 +0000 UTC m=+284.878127458" watchObservedRunningTime="2026-09-15 12:30:03.524725724 +0000 UTC m=+284.879572049" Sep 15 12:30:04.913460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:04.913419 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qp4rk"] Sep 15 12:30:04.916994 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:04.916970 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:04.919433 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:04.919410 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2sqc5\"" Sep 15 12:30:04.919584 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:04.919565 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 15 12:30:04.919701 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:04.919682 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 15 12:30:04.924415 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:04.924046 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qp4rk"] Sep 15 12:30:05.081542 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.081504 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5297c22-7938-4f32-a87a-99d14371d8e8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qp4rk\" (UID: \"c5297c22-7938-4f32-a87a-99d14371d8e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:05.081731 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.081549 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjt2w\" (UniqueName: \"kubernetes.io/projected/c5297c22-7938-4f32-a87a-99d14371d8e8-kube-api-access-vjt2w\") pod \"cert-manager-webhook-6486ffd997-qp4rk\" (UID: \"c5297c22-7938-4f32-a87a-99d14371d8e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:05.182743 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.182645 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5297c22-7938-4f32-a87a-99d14371d8e8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qp4rk\" (UID: \"c5297c22-7938-4f32-a87a-99d14371d8e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:05.182743 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.182694 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjt2w\" (UniqueName: \"kubernetes.io/projected/c5297c22-7938-4f32-a87a-99d14371d8e8-kube-api-access-vjt2w\") pod \"cert-manager-webhook-6486ffd997-qp4rk\" (UID: \"c5297c22-7938-4f32-a87a-99d14371d8e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:05.193207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.193178 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5297c22-7938-4f32-a87a-99d14371d8e8-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qp4rk\" (UID: \"c5297c22-7938-4f32-a87a-99d14371d8e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:05.193339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.193329 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjt2w\" (UniqueName: \"kubernetes.io/projected/c5297c22-7938-4f32-a87a-99d14371d8e8-kube-api-access-vjt2w\") pod \"cert-manager-webhook-6486ffd997-qp4rk\" (UID: \"c5297c22-7938-4f32-a87a-99d14371d8e8\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:05.242278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.242238 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:05.515021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:05.514996 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qp4rk"] Sep 15 12:30:05.517679 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:30:05.517653 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5297c22_7938_4f32_a87a_99d14371d8e8.slice/crio-873d149613f3298d3280903acdfa3aedeb1ca6f6dc1ccfba4ced0d38c2796f9a WatchSource:0}: Error finding container 873d149613f3298d3280903acdfa3aedeb1ca6f6dc1ccfba4ced0d38c2796f9a: Status 404 returned error can't find the container with id 873d149613f3298d3280903acdfa3aedeb1ca6f6dc1ccfba4ced0d38c2796f9a Sep 15 12:30:06.521802 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:06.521766 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" event={"ID":"c5297c22-7938-4f32-a87a-99d14371d8e8","Type":"ContainerStarted","Data":"873d149613f3298d3280903acdfa3aedeb1ca6f6dc1ccfba4ced0d38c2796f9a"} Sep 15 12:30:08.533809 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:08.533778 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" event={"ID":"c5297c22-7938-4f32-a87a-99d14371d8e8","Type":"ContainerStarted","Data":"82477b5cd1e4f38d8f75e938af70bc8e4db31d551a20e46996cf85d82716040d"} Sep 15 12:30:08.534197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:08.533904 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:08.549604 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:08.549551 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" podStartSLOduration=1.63179925 podStartE2EDuration="4.549533086s" totalImagesPullingTime="2.917733836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:08.548698375 +0000 UTC m=+289.903544699" watchObservedRunningTime="2026-09-15 12:30:08.549533086 +0000 UTC m=+289.904379408" Sep 15 12:30:14.540815 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:14.540784 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qp4rk" Sep 15 12:30:16.577491 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.577455 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pkj6s"] Sep 15 12:30:16.580710 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.580692 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:16.583375 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.583352 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kssmx\"" Sep 15 12:30:16.589547 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.589521 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pkj6s"] Sep 15 12:30:16.679498 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.679459 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcjms\" (UniqueName: \"kubernetes.io/projected/1fe44d1f-2d4b-4cad-8294-6f129719c1f5-kube-api-access-pcjms\") pod \"cert-manager-55b4d954d5-pkj6s\" (UID: \"1fe44d1f-2d4b-4cad-8294-6f129719c1f5\") " pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:16.679693 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.679507 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe44d1f-2d4b-4cad-8294-6f129719c1f5-bound-sa-token\") pod \"cert-manager-55b4d954d5-pkj6s\" (UID: \"1fe44d1f-2d4b-4cad-8294-6f129719c1f5\") " pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:16.780861 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.780815 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcjms\" (UniqueName: \"kubernetes.io/projected/1fe44d1f-2d4b-4cad-8294-6f129719c1f5-kube-api-access-pcjms\") pod \"cert-manager-55b4d954d5-pkj6s\" (UID: \"1fe44d1f-2d4b-4cad-8294-6f129719c1f5\") " pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:16.781047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.780875 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe44d1f-2d4b-4cad-8294-6f129719c1f5-bound-sa-token\") pod \"cert-manager-55b4d954d5-pkj6s\" (UID: \"1fe44d1f-2d4b-4cad-8294-6f129719c1f5\") " pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:16.789667 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.789631 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fe44d1f-2d4b-4cad-8294-6f129719c1f5-bound-sa-token\") pod \"cert-manager-55b4d954d5-pkj6s\" (UID: \"1fe44d1f-2d4b-4cad-8294-6f129719c1f5\") " pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:16.789866 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.789842 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcjms\" (UniqueName: \"kubernetes.io/projected/1fe44d1f-2d4b-4cad-8294-6f129719c1f5-kube-api-access-pcjms\") pod \"cert-manager-55b4d954d5-pkj6s\" (UID: \"1fe44d1f-2d4b-4cad-8294-6f129719c1f5\") " pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:16.890124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:16.890038 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pkj6s" Sep 15 12:30:17.160309 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:17.160285 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pkj6s"] Sep 15 12:30:17.162947 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:30:17.162918 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe44d1f_2d4b_4cad_8294_6f129719c1f5.slice/crio-fa6e6d1d305d6d88dc699225a3477cb2f21f05162f65f51cedfc36ccf7b664e4 WatchSource:0}: Error finding container fa6e6d1d305d6d88dc699225a3477cb2f21f05162f65f51cedfc36ccf7b664e4: Status 404 returned error can't find the container with id fa6e6d1d305d6d88dc699225a3477cb2f21f05162f65f51cedfc36ccf7b664e4 Sep 15 12:30:17.574427 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:17.574360 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pkj6s" event={"ID":"1fe44d1f-2d4b-4cad-8294-6f129719c1f5","Type":"ContainerStarted","Data":"35ee454d808c7c4f6ca4692e1fdea29b5e6431c4592c5c7d5852c588e00276d8"} Sep 15 12:30:17.574604 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:17.574437 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pkj6s" event={"ID":"1fe44d1f-2d4b-4cad-8294-6f129719c1f5","Type":"ContainerStarted","Data":"fa6e6d1d305d6d88dc699225a3477cb2f21f05162f65f51cedfc36ccf7b664e4"} Sep 15 12:30:17.590862 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:17.590810 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pkj6s" podStartSLOduration=1.590793475 podStartE2EDuration="1.590793475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:30:17.590119707 +0000 UTC m=+298.944966030" watchObservedRunningTime="2026-09-15 12:30:17.590793475 +0000 UTC m=+298.945639797" Sep 15 12:30:19.059530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:19.059499 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:30:19.064500 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:19.064467 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:30:19.071748 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:30:19.071721 2550 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 12:31:00.280285 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.280250 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j"] Sep 15 12:31:00.283446 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.283430 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.288833 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.288807 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 15 12:31:00.288833 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.288830 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 15 12:31:00.289068 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.289054 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 15 12:31:00.290200 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.290174 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 15 12:31:00.290303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.290198 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5mcvl\"" Sep 15 12:31:00.290303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.290250 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 15 12:31:00.306081 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.306047 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j"] Sep 15 12:31:00.358515 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.358477 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9082decf-f458-46bc-8520-c53854d47e6f-cert\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.358515 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.358514 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9082decf-f458-46bc-8520-c53854d47e6f-metrics-cert\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.358713 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.358540 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9082decf-f458-46bc-8520-c53854d47e6f-manager-config\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.358713 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.358603 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8nx\" (UniqueName: \"kubernetes.io/projected/9082decf-f458-46bc-8520-c53854d47e6f-kube-api-access-qc8nx\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.460012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.459974 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9082decf-f458-46bc-8520-c53854d47e6f-cert\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.460012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.460011 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9082decf-f458-46bc-8520-c53854d47e6f-metrics-cert\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.460246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.460041 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9082decf-f458-46bc-8520-c53854d47e6f-manager-config\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.460246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.460084 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8nx\" (UniqueName: \"kubernetes.io/projected/9082decf-f458-46bc-8520-c53854d47e6f-kube-api-access-qc8nx\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.460825 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.460798 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9082decf-f458-46bc-8520-c53854d47e6f-manager-config\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.462473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.462451 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9082decf-f458-46bc-8520-c53854d47e6f-metrics-cert\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.462676 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.462655 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9082decf-f458-46bc-8520-c53854d47e6f-cert\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.469233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.469205 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8nx\" (UniqueName: \"kubernetes.io/projected/9082decf-f458-46bc-8520-c53854d47e6f-kube-api-access-qc8nx\") pod \"lws-controller-manager-b47d9b998-v8d9j\" (UID: \"9082decf-f458-46bc-8520-c53854d47e6f\") " pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.592340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.592254 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:00.878542 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.878518 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j"] Sep 15 12:31:00.881179 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:31:00.881151 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9082decf_f458_46bc_8520_c53854d47e6f.slice/crio-674f06bb540e2f2c640f2891df08dce21878ffa4605f1db2b94e5f72206b0c9a WatchSource:0}: Error finding container 674f06bb540e2f2c640f2891df08dce21878ffa4605f1db2b94e5f72206b0c9a: Status 404 returned error can't find the container with id 674f06bb540e2f2c640f2891df08dce21878ffa4605f1db2b94e5f72206b0c9a Sep 15 12:31:00.883060 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:00.883045 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:31:01.766236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:01.766198 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" event={"ID":"9082decf-f458-46bc-8520-c53854d47e6f","Type":"ContainerStarted","Data":"674f06bb540e2f2c640f2891df08dce21878ffa4605f1db2b94e5f72206b0c9a"} Sep 15 12:31:03.777052 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:03.776963 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" event={"ID":"9082decf-f458-46bc-8520-c53854d47e6f","Type":"ContainerStarted","Data":"adf7029826a522db917ace4aac0971f48df8529b800a6e8c7ede2ec0599b0f84"} Sep 15 12:31:03.777052 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:03.777045 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:03.793133 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:03.793078 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" podStartSLOduration=1.182523799 podStartE2EDuration="3.79306393s" totalImagesPullingTime="2.610540131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:03.791445538 +0000 UTC m=+345.146291863" watchObservedRunningTime="2026-09-15 12:31:03.79306393 +0000 UTC m=+345.147910253" Sep 15 12:31:09.720887 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.720845 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886"] Sep 15 12:31:09.724536 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.724507 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.727094 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.727070 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 12:31:09.727621 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.727595 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k9sn2\"" Sep 15 12:31:09.727621 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.727613 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 12:31:09.727787 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.727625 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 12:31:09.728049 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.728032 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 12:31:09.746704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.746677 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886"] Sep 15 12:31:09.841219 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.841175 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9a6e984-a024-4371-be93-cf69c0653165-apiservice-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.841450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.841253 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9a6e984-a024-4371-be93-cf69c0653165-webhook-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.841450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.841287 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9a6e984-a024-4371-be93-cf69c0653165-manifests\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.841450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.841311 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj2ff\" (UniqueName: \"kubernetes.io/projected/b9a6e984-a024-4371-be93-cf69c0653165-kube-api-access-tj2ff\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.841450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.841399 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9a6e984-a024-4371-be93-cf69c0653165-tmp\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.942816 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.942772 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9a6e984-a024-4371-be93-cf69c0653165-webhook-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.942816 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.942820 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9a6e984-a024-4371-be93-cf69c0653165-manifests\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.943054 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.942839 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj2ff\" (UniqueName: \"kubernetes.io/projected/b9a6e984-a024-4371-be93-cf69c0653165-kube-api-access-tj2ff\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.943054 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.943031 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9a6e984-a024-4371-be93-cf69c0653165-tmp\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.943156 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.943132 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9a6e984-a024-4371-be93-cf69c0653165-apiservice-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.943208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.943179 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9a6e984-a024-4371-be93-cf69c0653165-manifests\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.945298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.945268 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9a6e984-a024-4371-be93-cf69c0653165-tmp\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.945451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.945337 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9a6e984-a024-4371-be93-cf69c0653165-webhook-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.945451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.945435 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9a6e984-a024-4371-be93-cf69c0653165-apiservice-cert\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:09.951159 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:09.951132 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj2ff\" (UniqueName: \"kubernetes.io/projected/b9a6e984-a024-4371-be93-cf69c0653165-kube-api-access-tj2ff\") pod \"opendatahub-operator-controller-manager-665cd58c4f-kc886\" (UID: \"b9a6e984-a024-4371-be93-cf69c0653165\") " pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:10.035347 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:10.035306 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:10.319023 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:10.318994 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886"] Sep 15 12:31:10.321794 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:31:10.321766 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a6e984_a024_4371_be93_cf69c0653165.slice/crio-f50286f944edeb243e982ecf8673bacaff0df5609da9fa2268bf9d18e244c2ca WatchSource:0}: Error finding container f50286f944edeb243e982ecf8673bacaff0df5609da9fa2268bf9d18e244c2ca: Status 404 returned error can't find the container with id f50286f944edeb243e982ecf8673bacaff0df5609da9fa2268bf9d18e244c2ca Sep 15 12:31:10.808266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:10.808232 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" event={"ID":"b9a6e984-a024-4371-be93-cf69c0653165","Type":"ContainerStarted","Data":"f50286f944edeb243e982ecf8673bacaff0df5609da9fa2268bf9d18e244c2ca"} Sep 15 12:31:14.783932 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:14.783894 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b47d9b998-v8d9j" Sep 15 12:31:14.836299 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:14.836263 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" event={"ID":"b9a6e984-a024-4371-be93-cf69c0653165","Type":"ContainerStarted","Data":"7437c3a24014a7fdac6841efed47f73da10a98e5740b71d3c22c5e96506a82a5"} Sep 15 12:31:15.842225 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:15.842190 2550 generic.go:356] "Generic (PLEG): container finished" podID="b9a6e984-a024-4371-be93-cf69c0653165" containerID="7437c3a24014a7fdac6841efed47f73da10a98e5740b71d3c22c5e96506a82a5" exitCode=0 Sep 15 12:31:15.842668 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:15.842281 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" event={"ID":"b9a6e984-a024-4371-be93-cf69c0653165","Type":"ContainerDied","Data":"7437c3a24014a7fdac6841efed47f73da10a98e5740b71d3c22c5e96506a82a5"} Sep 15 12:31:16.849890 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:16.849855 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" event={"ID":"b9a6e984-a024-4371-be93-cf69c0653165","Type":"ContainerStarted","Data":"f427546c6be9d66b28be98a31ff1195216d73871a9baf9f2b97109fefa04e9f2"} Sep 15 12:31:16.850260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:16.849992 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:16.871037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:16.870979 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" podStartSLOduration=2.79762595 podStartE2EDuration="7.870960561s" totalImagesPullingTime="3.972610559s" totalInitContainerRuntime="1.100724052s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:16.868980214 +0000 UTC m=+358.223826538" watchObservedRunningTime="2026-09-15 12:31:16.870960561 +0000 UTC m=+358.225806886" Sep 15 12:31:27.856820 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:27.856780 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-665cd58c4f-kc886" Sep 15 12:31:41.304271 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.304233 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt"] Sep 15 12:31:41.308471 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.308449 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.313281 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.311026 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ts8hb\"" Sep 15 12:31:41.313281 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.311253 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 12:31:41.320859 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.320831 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt"] Sep 15 12:31:41.412319 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412284 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412319 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412324 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412548 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412345 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412548 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412466 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412548 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412508 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412649 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412569 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412649 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412611 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4s5\" (UniqueName: \"kubernetes.io/projected/0db7cb94-84c4-444c-bdbf-58a3967843d1-kube-api-access-df4s5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412714 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412649 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db7cb94-84c4-444c-bdbf-58a3967843d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.412714 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.412686 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513190 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513158 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513199 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513258 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513281 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513307 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513346 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4s5\" (UniqueName: \"kubernetes.io/projected/0db7cb94-84c4-444c-bdbf-58a3967843d1-kube-api-access-df4s5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513726 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513406 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db7cb94-84c4-444c-bdbf-58a3967843d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513726 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513460 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513726 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513578 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513726 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513625 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513862 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513773 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.513862 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.513792 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.514124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.514106 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0db7cb94-84c4-444c-bdbf-58a3967843d1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.515682 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.515662 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.515917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.515898 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.524899 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.524878 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0db7cb94-84c4-444c-bdbf-58a3967843d1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.525014 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.524941 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4s5\" (UniqueName: \"kubernetes.io/projected/0db7cb94-84c4-444c-bdbf-58a3967843d1-kube-api-access-df4s5\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799ccldkt\" (UID: \"0db7cb94-84c4-444c-bdbf-58a3967843d1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.623119 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.623032 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:41.901218 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.901184 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt"] Sep 15 12:31:41.902968 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:31:41.902944 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db7cb94_84c4_444c_bdbf_58a3967843d1.slice/crio-60986796ec79dfcd07ffdd552a76ec0ab189fe1cd300be9bea2bf68c63690da4 WatchSource:0}: Error finding container 60986796ec79dfcd07ffdd552a76ec0ab189fe1cd300be9bea2bf68c63690da4: Status 404 returned error can't find the container with id 60986796ec79dfcd07ffdd552a76ec0ab189fe1cd300be9bea2bf68c63690da4 Sep 15 12:31:41.965169 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:41.965127 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" event={"ID":"0db7cb94-84c4-444c-bdbf-58a3967843d1","Type":"ContainerStarted","Data":"60986796ec79dfcd07ffdd552a76ec0ab189fe1cd300be9bea2bf68c63690da4"} Sep 15 12:31:42.651184 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.651150 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8gdgd"] Sep 15 12:31:42.655936 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.655908 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:42.658212 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.658184 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t6nnd\"" Sep 15 12:31:42.658354 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.658225 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 15 12:31:42.664963 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.664761 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8gdgd"] Sep 15 12:31:42.824544 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.824501 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkxt\" (UniqueName: \"kubernetes.io/projected/7bf764af-d105-4093-86ca-4613af06e0a3-kube-api-access-jkkxt\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:42.824713 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.824562 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:42.925246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.925161 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkxt\" (UniqueName: \"kubernetes.io/projected/7bf764af-d105-4093-86ca-4613af06e0a3-kube-api-access-jkkxt\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:42.925246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.925211 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:42.925470 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:31:42.925355 2550 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 15 12:31:42.925470 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:31:42.925431 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert podName:7bf764af-d105-4093-86ca-4613af06e0a3 nodeName:}" failed. No retries permitted until 2026-09-15 12:31:43.425411673 +0000 UTC m=+384.780257975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert") pod "odh-model-controller-7998c67dc4-8gdgd" (UID: "7bf764af-d105-4093-86ca-4613af06e0a3") : secret "odh-model-controller-webhook-cert" not found Sep 15 12:31:42.933826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:42.933801 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkxt\" (UniqueName: \"kubernetes.io/projected/7bf764af-d105-4093-86ca-4613af06e0a3-kube-api-access-jkkxt\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:43.085225 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.085194 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z9nkx"] Sep 15 12:31:43.088691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.088672 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.090917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.090898 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 15 12:31:43.091027 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.090900 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4r5dr\"" Sep 15 12:31:43.098681 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.098658 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z9nkx"] Sep 15 12:31:43.205265 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.205176 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cbd5594-5jsn9"] Sep 15 12:31:43.230521 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.228011 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c499b483-f74c-4e4b-84e9-e66a2b0941d8-tls-certs\") pod \"model-serving-api-54d89bd5fc-z9nkx\" (UID: \"c499b483-f74c-4e4b-84e9-e66a2b0941d8\") " pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.230521 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.228144 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4gt\" (UniqueName: \"kubernetes.io/projected/c499b483-f74c-4e4b-84e9-e66a2b0941d8-kube-api-access-js4gt\") pod \"model-serving-api-54d89bd5fc-z9nkx\" (UID: \"c499b483-f74c-4e4b-84e9-e66a2b0941d8\") " pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.328727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.328679 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4gt\" (UniqueName: \"kubernetes.io/projected/c499b483-f74c-4e4b-84e9-e66a2b0941d8-kube-api-access-js4gt\") pod \"model-serving-api-54d89bd5fc-z9nkx\" (UID: \"c499b483-f74c-4e4b-84e9-e66a2b0941d8\") " pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.328919 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.328781 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c499b483-f74c-4e4b-84e9-e66a2b0941d8-tls-certs\") pod \"model-serving-api-54d89bd5fc-z9nkx\" (UID: \"c499b483-f74c-4e4b-84e9-e66a2b0941d8\") " pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.331189 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.331170 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c499b483-f74c-4e4b-84e9-e66a2b0941d8-tls-certs\") pod \"model-serving-api-54d89bd5fc-z9nkx\" (UID: \"c499b483-f74c-4e4b-84e9-e66a2b0941d8\") " pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.336150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.336129 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4gt\" (UniqueName: \"kubernetes.io/projected/c499b483-f74c-4e4b-84e9-e66a2b0941d8-kube-api-access-js4gt\") pod \"model-serving-api-54d89bd5fc-z9nkx\" (UID: \"c499b483-f74c-4e4b-84e9-e66a2b0941d8\") " pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.400170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.400136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:43.429636 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.429602 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:43.429783 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:31:43.429750 2550 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 15 12:31:43.429832 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:31:43.429816 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert podName:7bf764af-d105-4093-86ca-4613af06e0a3 nodeName:}" failed. No retries permitted until 2026-09-15 12:31:44.429799963 +0000 UTC m=+385.784646265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert") pod "odh-model-controller-7998c67dc4-8gdgd" (UID: "7bf764af-d105-4093-86ca-4613af06e0a3") : secret "odh-model-controller-webhook-cert" not found Sep 15 12:31:43.669112 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.669081 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-z9nkx"] Sep 15 12:31:43.672369 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:31:43.672339 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc499b483_f74c_4e4b_84e9_e66a2b0941d8.slice/crio-7fa80b163dc856f356d52c336fd53f5ba0dc09accd5de23a3c919a85cc04137b WatchSource:0}: Error finding container 7fa80b163dc856f356d52c336fd53f5ba0dc09accd5de23a3c919a85cc04137b: Status 404 returned error can't find the container with id 7fa80b163dc856f356d52c336fd53f5ba0dc09accd5de23a3c919a85cc04137b Sep 15 12:31:43.976111 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:43.976030 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" event={"ID":"c499b483-f74c-4e4b-84e9-e66a2b0941d8","Type":"ContainerStarted","Data":"7fa80b163dc856f356d52c336fd53f5ba0dc09accd5de23a3c919a85cc04137b"} Sep 15 12:31:44.440251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:44.440209 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:44.443828 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:44.443798 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bf764af-d105-4093-86ca-4613af06e0a3-cert\") pod \"odh-model-controller-7998c67dc4-8gdgd\" (UID: \"7bf764af-d105-4093-86ca-4613af06e0a3\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:44.469279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:44.469241 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:44.758249 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:44.758219 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8gdgd"] Sep 15 12:31:44.761023 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:31:44.761000 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf764af_d105_4093_86ca_4613af06e0a3.slice/crio-1c725550361eec9e64ffd4b4db45ee5b46ba310c685b0dd5b84370b77746eb4e WatchSource:0}: Error finding container 1c725550361eec9e64ffd4b4db45ee5b46ba310c685b0dd5b84370b77746eb4e: Status 404 returned error can't find the container with id 1c725550361eec9e64ffd4b4db45ee5b46ba310c685b0dd5b84370b77746eb4e Sep 15 12:31:44.982705 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:44.982658 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" event={"ID":"7bf764af-d105-4093-86ca-4613af06e0a3","Type":"ContainerStarted","Data":"1c725550361eec9e64ffd4b4db45ee5b46ba310c685b0dd5b84370b77746eb4e"} Sep 15 12:31:46.995190 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:46.994858 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" event={"ID":"c499b483-f74c-4e4b-84e9-e66a2b0941d8","Type":"ContainerStarted","Data":"8cc6144ae97306a436decc10fe916c007d12462242ea3ef9e24bbb7e7b6b6213"} Sep 15 12:31:46.995190 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:46.994954 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:47.011917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:47.011854 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" podStartSLOduration=1.748039154 podStartE2EDuration="4.011839657s" totalImagesPullingTime="2.263800503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:47.010155322 +0000 UTC m=+388.365001648" watchObservedRunningTime="2026-09-15 12:31:47.011839657 +0000 UTC m=+388.366685981" Sep 15 12:31:49.006992 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.006956 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" event={"ID":"7bf764af-d105-4093-86ca-4613af06e0a3","Type":"ContainerStarted","Data":"e5a034669f9d38990d005ab218f0ac9448f8bcbd7939ddc347f8bb8ce0856818"} Sep 15 12:31:49.007368 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.007043 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:31:49.050831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.050780 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" podStartSLOduration=3.860248302 podStartE2EDuration="7.050763851s" totalImagesPullingTime="3.190515549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:49.050021183 +0000 UTC m=+390.404867510" watchObservedRunningTime="2026-09-15 12:31:49.050763851 +0000 UTC m=+390.405610173" Sep 15 12:31:49.303396 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.303295 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x"] Sep 15 12:31:49.307002 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.306977 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.309639 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.309618 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 15 12:31:49.310066 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.310051 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 15 12:31:49.310287 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.310255 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 15 12:31:49.310364 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.310304 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fwc8b\"" Sep 15 12:31:49.320371 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.320349 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x"] Sep 15 12:31:49.386917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.386879 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtnd\" (UniqueName: \"kubernetes.io/projected/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-kube-api-access-4gtnd\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.386917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.386919 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.387115 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.386941 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.387115 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.387049 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.387115 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.387092 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.387115 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.387111 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.387258 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.387129 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.488295 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.488247 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.488295 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.488296 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.488588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.488314 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.488588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.488342 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtnd\" (UniqueName: \"kubernetes.io/projected/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-kube-api-access-4gtnd\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.488588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.488360 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.488588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.488375 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.488588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.488499 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.489299 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.489272 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.490903 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.490875 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.491018 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.491001 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.491062 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.491020 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.491111 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.491095 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.501665 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.501641 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtnd\" (UniqueName: \"kubernetes.io/projected/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-kube-api-access-4gtnd\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.505362 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.505339 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6aba3fb-247c-4816-8b0a-c5f2c35c06cf-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nwd8x\" (UID: \"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.619123 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.619044 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:49.903891 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:49.903865 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x"] Sep 15 12:31:49.906236 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:31:49.906214 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aba3fb_247c_4816_8b0a_c5f2c35c06cf.slice/crio-5a0b2a8c409f332ff0fada61d218a2f1ce8f1e98c0afde7621e3007925ab0699 WatchSource:0}: Error finding container 5a0b2a8c409f332ff0fada61d218a2f1ce8f1e98c0afde7621e3007925ab0699: Status 404 returned error can't find the container with id 5a0b2a8c409f332ff0fada61d218a2f1ce8f1e98c0afde7621e3007925ab0699 Sep 15 12:31:50.013704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:50.013666 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" event={"ID":"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf","Type":"ContainerStarted","Data":"5a0b2a8c409f332ff0fada61d218a2f1ce8f1e98c0afde7621e3007925ab0699"} Sep 15 12:31:52.101872 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:52.101823 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:31:52.102146 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:52.101894 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:31:53.030642 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:53.030601 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" event={"ID":"f6aba3fb-247c-4816-8b0a-c5f2c35c06cf","Type":"ContainerStarted","Data":"38acc0140b0a8d368c973e0a87cba822efa421d47dd92cb7bfdc8a48b5c3a01d"} Sep 15 12:31:53.030838 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:53.030779 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:53.032668 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:53.032645 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" Sep 15 12:31:53.058993 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:53.058943 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nwd8x" podStartSLOduration=1.865636664 podStartE2EDuration="4.058926365s" totalImagesPullingTime="2.193289701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:53.056651741 +0000 UTC m=+394.411498064" watchObservedRunningTime="2026-09-15 12:31:53.058926365 +0000 UTC m=+394.413772688" Sep 15 12:31:56.141524 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:56.141483 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:31:56.141790 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:56.141557 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:31:56.141790 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:56.141586 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:31:57.061524 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:57.061492 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" event={"ID":"0db7cb94-84c4-444c-bdbf-58a3967843d1","Type":"ContainerStarted","Data":"9423fd0fc3f2a9defc6b0f04e8e9e04f663402627f39e5d2c64297d270a8d218"} Sep 15 12:31:57.080435 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:57.080368 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" podStartSLOduration=1.844167997 podStartE2EDuration="16.080353025s" totalImagesPullingTime="14.236185028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:31:57.078822558 +0000 UTC m=+398.433668882" watchObservedRunningTime="2026-09-15 12:31:57.080353025 +0000 UTC m=+398.435199347" Sep 15 12:31:57.623459 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:57.623419 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:57.627861 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:57.627837 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:58.005617 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:58.005590 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-z9nkx" Sep 15 12:31:58.066650 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:58.066615 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:31:58.067457 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:31:58.067429 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799ccldkt" Sep 15 12:32:00.016739 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:00.016710 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8gdgd" Sep 15 12:32:08.226982 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.226942 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66cbd5594-5jsn9" podUID="850d66e7-4177-47b9-a91d-3ab064adb273" containerName="console" containerID="cri-o://d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d" gracePeriod=15 Sep 15 12:32:08.466047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.466022 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:32:08.560541 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.560515 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-serving-cert\") pod \"850d66e7-4177-47b9-a91d-3ab064adb273\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " Sep 15 12:32:08.560718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.560557 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-trusted-ca-bundle\") pod \"850d66e7-4177-47b9-a91d-3ab064adb273\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " Sep 15 12:32:08.560718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.560580 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjgc5\" (UniqueName: \"kubernetes.io/projected/850d66e7-4177-47b9-a91d-3ab064adb273-kube-api-access-hjgc5\") pod \"850d66e7-4177-47b9-a91d-3ab064adb273\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " Sep 15 12:32:08.560718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.560594 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-console-config\") pod \"850d66e7-4177-47b9-a91d-3ab064adb273\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " Sep 15 12:32:08.560718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.560619 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-oauth-serving-cert\") pod \"850d66e7-4177-47b9-a91d-3ab064adb273\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " Sep 15 12:32:08.560718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.560659 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-service-ca\") pod \"850d66e7-4177-47b9-a91d-3ab064adb273\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " Sep 15 12:32:08.560718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.560693 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-oauth-config\") pod \"850d66e7-4177-47b9-a91d-3ab064adb273\" (UID: \"850d66e7-4177-47b9-a91d-3ab064adb273\") " Sep 15 12:32:08.561059 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.561026 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-console-config" (OuterVolumeSpecName: "console-config") pod "850d66e7-4177-47b9-a91d-3ab064adb273" (UID: "850d66e7-4177-47b9-a91d-3ab064adb273"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:32:08.561059 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.561040 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "850d66e7-4177-47b9-a91d-3ab064adb273" (UID: "850d66e7-4177-47b9-a91d-3ab064adb273"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:32:08.561174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.561051 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "850d66e7-4177-47b9-a91d-3ab064adb273" (UID: "850d66e7-4177-47b9-a91d-3ab064adb273"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:32:08.561174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.561096 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-service-ca" (OuterVolumeSpecName: "service-ca") pod "850d66e7-4177-47b9-a91d-3ab064adb273" (UID: "850d66e7-4177-47b9-a91d-3ab064adb273"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:32:08.562821 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.562792 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "850d66e7-4177-47b9-a91d-3ab064adb273" (UID: "850d66e7-4177-47b9-a91d-3ab064adb273"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:32:08.562913 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.562812 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850d66e7-4177-47b9-a91d-3ab064adb273-kube-api-access-hjgc5" (OuterVolumeSpecName: "kube-api-access-hjgc5") pod "850d66e7-4177-47b9-a91d-3ab064adb273" (UID: "850d66e7-4177-47b9-a91d-3ab064adb273"). InnerVolumeSpecName "kube-api-access-hjgc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:32:08.562913 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.562806 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "850d66e7-4177-47b9-a91d-3ab064adb273" (UID: "850d66e7-4177-47b9-a91d-3ab064adb273"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:32:08.661477 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.661445 2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:32:08.661477 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.661473 2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-oauth-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:32:08.661477 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.661484 2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850d66e7-4177-47b9-a91d-3ab064adb273-console-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:32:08.661691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.661494 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-trusted-ca-bundle\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:32:08.661691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.661502 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjgc5\" (UniqueName: \"kubernetes.io/projected/850d66e7-4177-47b9-a91d-3ab064adb273-kube-api-access-hjgc5\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:32:08.661691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.661512 2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-console-config\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:32:08.661691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:08.661520 2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850d66e7-4177-47b9-a91d-3ab064adb273-oauth-serving-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:32:09.119049 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.119017 2550 generic.go:356] "Generic (PLEG): container finished" podID="850d66e7-4177-47b9-a91d-3ab064adb273" containerID="d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d" exitCode=0 Sep 15 12:32:09.119199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.119060 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66cbd5594-5jsn9" Sep 15 12:32:09.119199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.119106 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cbd5594-5jsn9" event={"ID":"850d66e7-4177-47b9-a91d-3ab064adb273","Type":"ContainerDied","Data":"d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d"} Sep 15 12:32:09.119199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.119157 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66cbd5594-5jsn9" event={"ID":"850d66e7-4177-47b9-a91d-3ab064adb273","Type":"ContainerDied","Data":"bff7248850ec6947084ed3681f7fc995cefc777b36c1e1edbc325dfb7917369f"} Sep 15 12:32:09.119199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.119184 2550 scope.go:117] "RemoveContainer" containerID="d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d" Sep 15 12:32:09.128791 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.128774 2550 scope.go:117] "RemoveContainer" containerID="d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d" Sep 15 12:32:09.129069 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:32:09.129050 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d\": container with ID starting with d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d not found: ID does not exist" containerID="d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d" Sep 15 12:32:09.129111 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.129077 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d"} err="failed to get container status \"d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d\": rpc error: code = NotFound desc = could not find container \"d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d\": container with ID starting with d225c2e177b8b8d54fce1cc7d437186bec085841c1b508048b1dc47101cf3a3d not found: ID does not exist" Sep 15 12:32:09.140001 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.139977 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66cbd5594-5jsn9"] Sep 15 12:32:09.144738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.144717 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66cbd5594-5jsn9"] Sep 15 12:32:09.154537 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:32:09.154515 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850d66e7-4177-47b9-a91d-3ab064adb273" path="/var/lib/kubelet/pods/850d66e7-4177-47b9-a91d-3ab064adb273/volumes" Sep 15 12:35:19.091703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:19.091677 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:35:19.100028 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:19.100006 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:35:39.841866 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.841826 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-v9pnn"] Sep 15 12:35:39.842328 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.842189 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850d66e7-4177-47b9-a91d-3ab064adb273" containerName="console" Sep 15 12:35:39.842328 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.842201 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="850d66e7-4177-47b9-a91d-3ab064adb273" containerName="console" Sep 15 12:35:39.842328 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.842282 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="850d66e7-4177-47b9-a91d-3ab064adb273" containerName="console" Sep 15 12:35:39.845430 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.845409 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:39.847816 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.847793 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 15 12:35:39.847886 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.847850 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fr74p\"" Sep 15 12:35:39.857279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:39.857254 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-v9pnn"] Sep 15 12:35:40.015047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.015010 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krfwp\" (UniqueName: \"kubernetes.io/projected/107e94e7-82fd-4fda-a35e-28b1d35c5b2b-kube-api-access-krfwp\") pod \"payload-processing-5b54b7fbb7-v9pnn\" (UID: \"107e94e7-82fd-4fda-a35e-28b1d35c5b2b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:40.015224 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.015081 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e94e7-82fd-4fda-a35e-28b1d35c5b2b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-v9pnn\" (UID: \"107e94e7-82fd-4fda-a35e-28b1d35c5b2b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:40.115986 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.115914 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krfwp\" (UniqueName: \"kubernetes.io/projected/107e94e7-82fd-4fda-a35e-28b1d35c5b2b-kube-api-access-krfwp\") pod \"payload-processing-5b54b7fbb7-v9pnn\" (UID: \"107e94e7-82fd-4fda-a35e-28b1d35c5b2b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:40.115986 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.115969 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e94e7-82fd-4fda-a35e-28b1d35c5b2b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-v9pnn\" (UID: \"107e94e7-82fd-4fda-a35e-28b1d35c5b2b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:40.116620 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.116601 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/107e94e7-82fd-4fda-a35e-28b1d35c5b2b-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-v9pnn\" (UID: \"107e94e7-82fd-4fda-a35e-28b1d35c5b2b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:40.126571 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.126541 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krfwp\" (UniqueName: \"kubernetes.io/projected/107e94e7-82fd-4fda-a35e-28b1d35c5b2b-kube-api-access-krfwp\") pod \"payload-processing-5b54b7fbb7-v9pnn\" (UID: \"107e94e7-82fd-4fda-a35e-28b1d35c5b2b\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:40.156104 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.156060 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:40.424516 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:40.424492 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-v9pnn"] Sep 15 12:35:40.426303 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:35:40.426279 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107e94e7_82fd_4fda_a35e_28b1d35c5b2b.slice/crio-e6ce1e3a2d224bb46ba71293e5beff5f96cbaa312b2bb7865476774bcf281a57 WatchSource:0}: Error finding container e6ce1e3a2d224bb46ba71293e5beff5f96cbaa312b2bb7865476774bcf281a57: Status 404 returned error can't find the container with id e6ce1e3a2d224bb46ba71293e5beff5f96cbaa312b2bb7865476774bcf281a57 Sep 15 12:35:41.101619 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:41.101582 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" event={"ID":"107e94e7-82fd-4fda-a35e-28b1d35c5b2b","Type":"ContainerStarted","Data":"e6ce1e3a2d224bb46ba71293e5beff5f96cbaa312b2bb7865476774bcf281a57"} Sep 15 12:35:45.125301 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:45.125202 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" event={"ID":"107e94e7-82fd-4fda-a35e-28b1d35c5b2b","Type":"ContainerStarted","Data":"2a18bd4e8fc95e9855cd7977b80519392140bf75c9e735e83539d6af3282a2d1"} Sep 15 12:35:45.125767 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:45.125346 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:35:45.141660 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:45.141598 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" podStartSLOduration=1.8248256280000001 podStartE2EDuration="6.141583008s" totalImagesPullingTime="4.31675738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:35:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:35:45.140460247 +0000 UTC m=+626.495306571" watchObservedRunningTime="2026-09-15 12:35:45.141583008 +0000 UTC m=+626.496429330" Sep 15 12:35:56.132801 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:35:56.132769 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-v9pnn" Sep 15 12:36:15.146139 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.146102 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl"] Sep 15 12:36:15.149502 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.149480 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.152815 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.152792 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 12:36:15.152976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.152805 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 12:36:15.153049 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.152965 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2598m\"" Sep 15 12:36:15.153214 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.153197 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 15 12:36:15.159568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.159546 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl"] Sep 15 12:36:15.233266 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.233229 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.233468 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.233285 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.233468 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.233355 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.233468 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.233441 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.233603 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.233469 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dhw\" (UniqueName: \"kubernetes.io/projected/f15029f5-a5c3-475a-97ab-eacfe278f40c-kube-api-access-z2dhw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.233603 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.233504 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15029f5-a5c3-475a-97ab-eacfe278f40c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334249 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334218 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334267 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334288 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334323 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334339 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dhw\" (UniqueName: \"kubernetes.io/projected/f15029f5-a5c3-475a-97ab-eacfe278f40c-kube-api-access-z2dhw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334362 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15029f5-a5c3-475a-97ab-eacfe278f40c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334754 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334713 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334754 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334732 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.334905 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.334772 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.336672 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.336650 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f15029f5-a5c3-475a-97ab-eacfe278f40c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.336837 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.336819 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f15029f5-a5c3-475a-97ab-eacfe278f40c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.341767 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.341745 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dhw\" (UniqueName: \"kubernetes.io/projected/f15029f5-a5c3-475a-97ab-eacfe278f40c-kube-api-access-z2dhw\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-857wl\" (UID: \"f15029f5-a5c3-475a-97ab-eacfe278f40c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.461991 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.461907 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:15.730608 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.730579 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl"] Sep 15 12:36:15.733067 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:36:15.733037 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15029f5_a5c3_475a_97ab_eacfe278f40c.slice/crio-5696e61bf1ca5d583f38df152379922b78b3a7ea41bf7b2fe514cf71aa77f1f5 WatchSource:0}: Error finding container 5696e61bf1ca5d583f38df152379922b78b3a7ea41bf7b2fe514cf71aa77f1f5: Status 404 returned error can't find the container with id 5696e61bf1ca5d583f38df152379922b78b3a7ea41bf7b2fe514cf71aa77f1f5 Sep 15 12:36:15.734485 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:15.734465 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:36:16.272078 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:16.272038 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" event={"ID":"f15029f5-a5c3-475a-97ab-eacfe278f40c","Type":"ContainerStarted","Data":"5696e61bf1ca5d583f38df152379922b78b3a7ea41bf7b2fe514cf71aa77f1f5"} Sep 15 12:36:18.284078 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:18.284036 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" event={"ID":"f15029f5-a5c3-475a-97ab-eacfe278f40c","Type":"ContainerStarted","Data":"520460635a7337705b0814726686ec33acdb7995541d88e61e9ac10d280d108c"} Sep 15 12:36:18.302128 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:18.302076 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" podStartSLOduration=1.814897587 podStartE2EDuration="3.302061339s" totalImagesPullingTime="1.487163752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:36:18.300970944 +0000 UTC m=+659.655817264" watchObservedRunningTime="2026-09-15 12:36:18.302061339 +0000 UTC m=+659.656907662" Sep 15 12:36:20.555108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.555068 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv"] Sep 15 12:36:20.558848 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.558825 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.561031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.561008 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 15 12:36:20.569983 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.569960 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv"] Sep 15 12:36:20.686828 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.686788 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.686828 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.686824 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.687079 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.686842 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.687079 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.686861 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28e55bc-4437-4be7-a364-7afa0dbd9067-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.687079 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.686924 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.687079 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.687011 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s567r\" (UniqueName: \"kubernetes.io/projected/a28e55bc-4437-4be7-a364-7afa0dbd9067-kube-api-access-s567r\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788122 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788170 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788291 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788325 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28e55bc-4437-4be7-a364-7afa0dbd9067-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788353 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788428 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s567r\" (UniqueName: \"kubernetes.io/projected/a28e55bc-4437-4be7-a364-7afa0dbd9067-kube-api-access-s567r\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788728 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788696 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788796 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788723 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.788796 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.788753 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.790551 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.790524 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a28e55bc-4437-4be7-a364-7afa0dbd9067-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.790709 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.790692 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a28e55bc-4437-4be7-a364-7afa0dbd9067-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.795917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.795896 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s567r\" (UniqueName: \"kubernetes.io/projected/a28e55bc-4437-4be7-a364-7afa0dbd9067-kube-api-access-s567r\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v8tcv\" (UID: \"a28e55bc-4437-4be7-a364-7afa0dbd9067\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:20.869505 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:20.869425 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:21.140221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:21.140150 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv"] Sep 15 12:36:21.143202 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:36:21.143166 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda28e55bc_4437_4be7_a364_7afa0dbd9067.slice/crio-9ddfc18bb94750d9c8a6d381cd4cbbc63d004edefea689963f79cfa81a93c6bd WatchSource:0}: Error finding container 9ddfc18bb94750d9c8a6d381cd4cbbc63d004edefea689963f79cfa81a93c6bd: Status 404 returned error can't find the container with id 9ddfc18bb94750d9c8a6d381cd4cbbc63d004edefea689963f79cfa81a93c6bd Sep 15 12:36:21.299035 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:21.299000 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" event={"ID":"a28e55bc-4437-4be7-a364-7afa0dbd9067","Type":"ContainerStarted","Data":"9ddfc18bb94750d9c8a6d381cd4cbbc63d004edefea689963f79cfa81a93c6bd"} Sep 15 12:36:22.305923 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:22.305891 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" event={"ID":"a28e55bc-4437-4be7-a364-7afa0dbd9067","Type":"ContainerStarted","Data":"e62576602516136f3567983ff78d31bb999ab7cc9e5326b46075c9ff59285f23"} Sep 15 12:36:22.327363 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:22.327312 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" podStartSLOduration=2.114354632 podStartE2EDuration="2.327295833s" totalImagesPullingTime="212.941201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:36:22.325784244 +0000 UTC m=+663.680630567" watchObservedRunningTime="2026-09-15 12:36:22.327295833 +0000 UTC m=+663.682142156" Sep 15 12:36:25.462323 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:25.462282 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:25.462735 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:25.462343 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:25.474641 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:25.474609 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:26.336075 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:26.336048 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-857wl" Sep 15 12:36:30.869664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:30.869618 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:30.870128 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:30.869721 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:30.882363 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:30.882330 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:36:31.362285 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:36:31.362248 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v8tcv" Sep 15 12:38:25.062943 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.062907 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8"] Sep 15 12:38:25.067104 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.067081 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.069579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.069560 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-11f865-openshift-default-dockercfg-stb22\"" Sep 15 12:38:25.078278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.078251 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8"] Sep 15 12:38:25.111791 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.111757 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istiod-ca-cert\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.111998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.111798 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-token\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.111998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.111828 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-data\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.111998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.111902 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-envoy\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.111998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.111943 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-podinfo\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.112217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.112020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-credential-socket\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.112217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.112062 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-certs\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.112217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.112088 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jz6\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-kube-api-access-x4jz6\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.112217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.112129 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-socket\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.213683 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213640 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-credential-socket\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.213866 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213696 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-certs\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.213866 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213725 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jz6\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-kube-api-access-x4jz6\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.213866 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-socket\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.213866 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213816 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istiod-ca-cert\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.213866 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213854 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-token\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213898 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-data\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213948 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-envoy\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.213976 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-podinfo\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.214187 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-certs\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.214213 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-socket\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214570 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.214549 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istiod-ca-cert\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.214642 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-data\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.214719 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.214682 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-credential-socket\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.216595 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.216562 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-envoy\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.217131 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.217100 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-podinfo\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.221587 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.221565 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-token\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.222721 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.222695 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jz6\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-kube-api-access-x4jz6\") pod \"e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.379973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.379884 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:25.654142 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.654114 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8"] Sep 15 12:38:25.654868 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:25.654844 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a5d410_69b2_4ae2_8be3_a9b6e5a1c9f3.slice/crio-5acdc65ae667606b9fc643c922a8fc3165534847aaffd6d3ecb3bf871e595c80 WatchSource:0}: Error finding container 5acdc65ae667606b9fc643c922a8fc3165534847aaffd6d3ecb3bf871e595c80: Status 404 returned error can't find the container with id 5acdc65ae667606b9fc643c922a8fc3165534847aaffd6d3ecb3bf871e595c80 Sep 15 12:38:25.657067 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.657032 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:38:25.657145 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.657099 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:38:25.657145 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.657126 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:38:25.918105 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.918008 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" event={"ID":"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3","Type":"ContainerStarted","Data":"3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c"} Sep 15 12:38:25.918105 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.918088 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" event={"ID":"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3","Type":"ContainerStarted","Data":"5acdc65ae667606b9fc643c922a8fc3165534847aaffd6d3ecb3bf871e595c80"} Sep 15 12:38:25.936442 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:25.936366 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podStartSLOduration=0.936349668 podStartE2EDuration="936.349668ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:25.935436275 +0000 UTC m=+787.290282615" watchObservedRunningTime="2026-09-15 12:38:25.936349668 +0000 UTC m=+787.291195990" Sep 15 12:38:26.380575 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:26.380539 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:26.385684 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:26.385659 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:26.924057 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:26.924025 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:26.925019 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:26.924999 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:30.113842 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.113806 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz"] Sep 15 12:38:30.117416 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.117374 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.119607 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.119588 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-11f865\"" Sep 15 12:38:30.129509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.129481 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz"] Sep 15 12:38:30.158546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.158509 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw"] Sep 15 12:38:30.162094 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.162071 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.164555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.164531 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-11f865-dockercfg-rmh6h\"" Sep 15 12:38:30.172289 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.172267 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw"] Sep 15 12:38:30.262977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.262943 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1777c2-a59b-497e-b179-a8f4785d56a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.263166 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.263008 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34dfb603-5cee-478b-9fe1-cec070ff2e18-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.263166 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.263084 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mtj\" (UniqueName: \"kubernetes.io/projected/34dfb603-5cee-478b-9fe1-cec070ff2e18-kube-api-access-m5mtj\") pod \"payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.263166 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.263119 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99j4r\" (UniqueName: \"kubernetes.io/projected/2a1777c2-a59b-497e-b179-a8f4785d56a8-kube-api-access-99j4r\") pod \"payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.364527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.364439 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34dfb603-5cee-478b-9fe1-cec070ff2e18-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.364527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.364490 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mtj\" (UniqueName: \"kubernetes.io/projected/34dfb603-5cee-478b-9fe1-cec070ff2e18-kube-api-access-m5mtj\") pod \"payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.364527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.364516 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99j4r\" (UniqueName: \"kubernetes.io/projected/2a1777c2-a59b-497e-b179-a8f4785d56a8-kube-api-access-99j4r\") pod \"payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.364812 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.364635 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1777c2-a59b-497e-b179-a8f4785d56a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.365278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.365251 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34dfb603-5cee-478b-9fe1-cec070ff2e18-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.365364 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.365298 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1777c2-a59b-497e-b179-a8f4785d56a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.373281 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.373258 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99j4r\" (UniqueName: \"kubernetes.io/projected/2a1777c2-a59b-497e-b179-a8f4785d56a8-kube-api-access-99j4r\") pod \"payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.373365 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.373319 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mtj\" (UniqueName: \"kubernetes.io/projected/34dfb603-5cee-478b-9fe1-cec070ff2e18-kube-api-access-m5mtj\") pod \"payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.429474 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.429434 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.473240 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.473202 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.702653 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.702622 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz"] Sep 15 12:38:30.704322 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:30.704296 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1777c2_a59b_497e_b179_a8f4785d56a8.slice/crio-43be24fddf2039e6bee818c6fb2d6b1d564bc802e71c49ea809ec4808467222c WatchSource:0}: Error finding container 43be24fddf2039e6bee818c6fb2d6b1d564bc802e71c49ea809ec4808467222c: Status 404 returned error can't find the container with id 43be24fddf2039e6bee818c6fb2d6b1d564bc802e71c49ea809ec4808467222c Sep 15 12:38:30.756000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.755974 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw"] Sep 15 12:38:30.758099 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:30.758076 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dfb603_5cee_478b_9fe1_cec070ff2e18.slice/crio-b9fde5e0a10cb5cfa8ac2c71d7f2e13364f8d75c8476fbcd741306fe87e8bd26 WatchSource:0}: Error finding container b9fde5e0a10cb5cfa8ac2c71d7f2e13364f8d75c8476fbcd741306fe87e8bd26: Status 404 returned error can't find the container with id b9fde5e0a10cb5cfa8ac2c71d7f2e13364f8d75c8476fbcd741306fe87e8bd26 Sep 15 12:38:30.949106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.949069 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" event={"ID":"34dfb603-5cee-478b-9fe1-cec070ff2e18","Type":"ContainerStarted","Data":"1891d26b0b817444c8ee46e8aa5332487fa29ecb2d2fea0364a4182aff4c89a3"} Sep 15 12:38:30.949304 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.949135 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" event={"ID":"34dfb603-5cee-478b-9fe1-cec070ff2e18","Type":"ContainerStarted","Data":"b9fde5e0a10cb5cfa8ac2c71d7f2e13364f8d75c8476fbcd741306fe87e8bd26"} Sep 15 12:38:30.949304 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.949292 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:30.950717 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.950693 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" event={"ID":"2a1777c2-a59b-497e-b179-a8f4785d56a8","Type":"ContainerStarted","Data":"fc78f9084f894c330efce8a61dfb11d510cd25c858652c7ebbe8b306049b84cf"} Sep 15 12:38:30.950840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.950732 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" event={"ID":"2a1777c2-a59b-497e-b179-a8f4785d56a8","Type":"ContainerStarted","Data":"43be24fddf2039e6bee818c6fb2d6b1d564bc802e71c49ea809ec4808467222c"} Sep 15 12:38:30.950840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.950810 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:30.966371 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.966255 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" podStartSLOduration=0.966233444 podStartE2EDuration="966.233444ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:30.963565121 +0000 UTC m=+792.318411447" watchObservedRunningTime="2026-09-15 12:38:30.966233444 +0000 UTC m=+792.321079776" Sep 15 12:38:30.977527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:30.977474 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" podStartSLOduration=0.977458574 podStartE2EDuration="977.458574ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:30.976819232 +0000 UTC m=+792.331665570" watchObservedRunningTime="2026-09-15 12:38:30.977458574 +0000 UTC m=+792.332304896" Sep 15 12:38:32.088275 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.088245 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh"] Sep 15 12:38:32.091870 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.091849 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.094341 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.094323 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ee96ec\"" Sep 15 12:38:32.094461 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.094409 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ee96ec-dockercfg-6xn4d\"" Sep 15 12:38:32.104712 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.104687 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh"] Sep 15 12:38:32.120974 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.120944 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk"] Sep 15 12:38:32.124281 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.124264 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.134187 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.134164 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk"] Sep 15 12:38:32.184180 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.184130 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139bd5e4-11c1-4561-a678-0f1322d3c5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.184342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.184230 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5b6l\" (UniqueName: \"kubernetes.io/projected/139bd5e4-11c1-4561-a678-0f1322d3c5a8-kube-api-access-v5b6l\") pod \"payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.285279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.285241 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.285279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.285281 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8gv\" (UniqueName: \"kubernetes.io/projected/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-kube-api-access-lx8gv\") pod \"payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.285510 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.285326 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139bd5e4-11c1-4561-a678-0f1322d3c5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.285510 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.285375 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5b6l\" (UniqueName: \"kubernetes.io/projected/139bd5e4-11c1-4561-a678-0f1322d3c5a8-kube-api-access-v5b6l\") pod \"payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.286025 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.286005 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139bd5e4-11c1-4561-a678-0f1322d3c5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.292584 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.292558 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5b6l\" (UniqueName: \"kubernetes.io/projected/139bd5e4-11c1-4561-a678-0f1322d3c5a8-kube-api-access-v5b6l\") pod \"payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.386154 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.386047 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.386154 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.386110 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8gv\" (UniqueName: \"kubernetes.io/projected/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-kube-api-access-lx8gv\") pod \"payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.387109 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.387084 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.393671 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.393649 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8gv\" (UniqueName: \"kubernetes.io/projected/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-kube-api-access-lx8gv\") pod \"payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.402512 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.402488 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.434899 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.434852 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.678273 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.678246 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh"] Sep 15 12:38:32.680583 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:32.680556 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139bd5e4_11c1_4561_a678_0f1322d3c5a8.slice/crio-64fb797de48a93a3e47e124f0f6b32675d12b8edbbbcc746bed60e838126a7f7 WatchSource:0}: Error finding container 64fb797de48a93a3e47e124f0f6b32675d12b8edbbbcc746bed60e838126a7f7: Status 404 returned error can't find the container with id 64fb797de48a93a3e47e124f0f6b32675d12b8edbbbcc746bed60e838126a7f7 Sep 15 12:38:32.723627 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.723603 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk"] Sep 15 12:38:32.726012 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:32.725888 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b2453a_58d8_4eaa_8f2e_9e2075dd695b.slice/crio-90f46ecd1b0711c30524800df11844ffd612af3d3a504e1f953f30fe78726929 WatchSource:0}: Error finding container 90f46ecd1b0711c30524800df11844ffd612af3d3a504e1f953f30fe78726929: Status 404 returned error can't find the container with id 90f46ecd1b0711c30524800df11844ffd612af3d3a504e1f953f30fe78726929 Sep 15 12:38:32.966684 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.966501 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" event={"ID":"139bd5e4-11c1-4561-a678-0f1322d3c5a8","Type":"ContainerStarted","Data":"88ec35cdfffe2a0b6ef6dcb74be65dda8b2c0919488055d27e3421402dc39426"} Sep 15 12:38:32.966684 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.966569 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" event={"ID":"139bd5e4-11c1-4561-a678-0f1322d3c5a8","Type":"ContainerStarted","Data":"64fb797de48a93a3e47e124f0f6b32675d12b8edbbbcc746bed60e838126a7f7"} Sep 15 12:38:32.966684 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.966601 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:32.968298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.968271 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" event={"ID":"72b2453a-58d8-4eaa-8f2e-9e2075dd695b","Type":"ContainerStarted","Data":"f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2"} Sep 15 12:38:32.968463 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.968315 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" event={"ID":"72b2453a-58d8-4eaa-8f2e-9e2075dd695b","Type":"ContainerStarted","Data":"90f46ecd1b0711c30524800df11844ffd612af3d3a504e1f953f30fe78726929"} Sep 15 12:38:32.968463 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.968345 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:32.975180 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.975157 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz"] Sep 15 12:38:32.978618 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.978602 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:32.980789 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.980763 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4609ba\"" Sep 15 12:38:32.980891 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.980812 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4609ba-dockercfg-6fgbw\"" Sep 15 12:38:32.988471 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.988451 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz"] Sep 15 12:38:32.990825 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:32.990775 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" podStartSLOduration=0.990758626 podStartE2EDuration="990.758626ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:32.989517265 +0000 UTC m=+794.344363588" watchObservedRunningTime="2026-09-15 12:38:32.990758626 +0000 UTC m=+794.345604944" Sep 15 12:38:33.007994 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.007969 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj"] Sep 15 12:38:33.011653 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.011638 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.015860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.015822 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" podStartSLOduration=1.01580908 podStartE2EDuration="1.01580908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:33.015231648 +0000 UTC m=+794.370077975" watchObservedRunningTime="2026-09-15 12:38:33.01580908 +0000 UTC m=+794.370655403" Sep 15 12:38:33.022072 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.022051 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj"] Sep 15 12:38:33.092409 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.092357 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6gc\" (UniqueName: \"kubernetes.io/projected/d4618891-5a99-45ef-96e6-021c7806cb2b-kube-api-access-st6gc\") pod \"payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.092863 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.092488 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4618891-5a99-45ef-96e6-021c7806cb2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.092863 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.092540 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjtm\" (UniqueName: \"kubernetes.io/projected/cca36773-1899-401b-bdb7-b311309903ad-kube-api-access-kvjtm\") pod \"payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.092863 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.092563 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca36773-1899-401b-bdb7-b311309903ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.193775 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.193735 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6gc\" (UniqueName: \"kubernetes.io/projected/d4618891-5a99-45ef-96e6-021c7806cb2b-kube-api-access-st6gc\") pod \"payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.193775 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.193776 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4618891-5a99-45ef-96e6-021c7806cb2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.194006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.193832 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjtm\" (UniqueName: \"kubernetes.io/projected/cca36773-1899-401b-bdb7-b311309903ad-kube-api-access-kvjtm\") pod \"payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.194006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.193870 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca36773-1899-401b-bdb7-b311309903ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.194757 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.194733 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4618891-5a99-45ef-96e6-021c7806cb2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.194822 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.194772 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca36773-1899-401b-bdb7-b311309903ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.206530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.206501 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6gc\" (UniqueName: \"kubernetes.io/projected/d4618891-5a99-45ef-96e6-021c7806cb2b-kube-api-access-st6gc\") pod \"payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.206699 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.206668 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjtm\" (UniqueName: \"kubernetes.io/projected/cca36773-1899-401b-bdb7-b311309903ad-kube-api-access-kvjtm\") pod \"payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.291667 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.291629 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.322831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.322802 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.569098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.568769 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz"] Sep 15 12:38:33.572464 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:33.572432 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca36773_1899_401b_bdb7_b311309903ad.slice/crio-d1e71923677ff063bd9d46d15f8c11ab9156316217ebc28a13b5ba98cb495700 WatchSource:0}: Error finding container d1e71923677ff063bd9d46d15f8c11ab9156316217ebc28a13b5ba98cb495700: Status 404 returned error can't find the container with id d1e71923677ff063bd9d46d15f8c11ab9156316217ebc28a13b5ba98cb495700 Sep 15 12:38:33.625795 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.625769 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj"] Sep 15 12:38:33.627477 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:33.627452 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4618891_5a99_45ef_96e6_021c7806cb2b.slice/crio-ec8ffef0b2be8d02c551afe7be2d8e6c3dcf0b471a0381cc5f191374ae1e612c WatchSource:0}: Error finding container ec8ffef0b2be8d02c551afe7be2d8e6c3dcf0b471a0381cc5f191374ae1e612c: Status 404 returned error can't find the container with id ec8ffef0b2be8d02c551afe7be2d8e6c3dcf0b471a0381cc5f191374ae1e612c Sep 15 12:38:33.978026 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.977922 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" event={"ID":"cca36773-1899-401b-bdb7-b311309903ad","Type":"ContainerStarted","Data":"8aedc89daf3d3c87d7486f04730317a9069ce0cbabae45c6e55578e4a734c9b5"} Sep 15 12:38:33.978212 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.978027 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" event={"ID":"cca36773-1899-401b-bdb7-b311309903ad","Type":"ContainerStarted","Data":"d1e71923677ff063bd9d46d15f8c11ab9156316217ebc28a13b5ba98cb495700"} Sep 15 12:38:33.978212 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.978197 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:33.979741 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.979714 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" event={"ID":"d4618891-5a99-45ef-96e6-021c7806cb2b","Type":"ContainerStarted","Data":"42aba35bd12fd4782bbdbfaf620b9569e665a6f69c40578a11d318737667ccf3"} Sep 15 12:38:33.979839 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.979762 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" event={"ID":"d4618891-5a99-45ef-96e6-021c7806cb2b","Type":"ContainerStarted","Data":"ec8ffef0b2be8d02c551afe7be2d8e6c3dcf0b471a0381cc5f191374ae1e612c"} Sep 15 12:38:33.979839 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.979815 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:33.995119 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:33.995066 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" podStartSLOduration=1.995052001 podStartE2EDuration="1.995052001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:33.993859856 +0000 UTC m=+795.348706184" watchObservedRunningTime="2026-09-15 12:38:33.995052001 +0000 UTC m=+795.349898324" Sep 15 12:38:34.013472 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:34.013423 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" podStartSLOduration=2.013406174 podStartE2EDuration="2.013406174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:34.011296449 +0000 UTC m=+795.366142774" watchObservedRunningTime="2026-09-15 12:38:34.013406174 +0000 UTC m=+795.368252497" Sep 15 12:38:41.278229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.278151 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc"] Sep 15 12:38:41.282891 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.282869 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.285109 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.285091 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-bf37f4\"" Sep 15 12:38:41.285208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.285141 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-bf37f4-dockercfg-2mmwx\"" Sep 15 12:38:41.289579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.289357 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc"] Sep 15 12:38:41.360685 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.360647 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6"] Sep 15 12:38:41.365073 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.365040 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.373719 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.373697 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6"] Sep 15 12:38:41.382211 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.382177 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b3e809-4391-48b7-845d-796a3f1d5965-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.382317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.382237 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzrs\" (UniqueName: \"kubernetes.io/projected/c3b3e809-4391-48b7-845d-796a3f1d5965-kube-api-access-qbzrs\") pod \"payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.483037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.483001 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc06679e-69e2-4121-9a9d-3b6556a48040-plugins-config-volume\") pod \"payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.483221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.483048 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b3e809-4391-48b7-845d-796a3f1d5965-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.483221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.483122 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzrs\" (UniqueName: \"kubernetes.io/projected/c3b3e809-4391-48b7-845d-796a3f1d5965-kube-api-access-qbzrs\") pod \"payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.483221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.483176 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75jt\" (UniqueName: \"kubernetes.io/projected/fc06679e-69e2-4121-9a9d-3b6556a48040-kube-api-access-b75jt\") pod \"payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.483829 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.483809 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b3e809-4391-48b7-845d-796a3f1d5965-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.491717 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.491691 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzrs\" (UniqueName: \"kubernetes.io/projected/c3b3e809-4391-48b7-845d-796a3f1d5965-kube-api-access-qbzrs\") pod \"payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.584844 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.584738 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75jt\" (UniqueName: \"kubernetes.io/projected/fc06679e-69e2-4121-9a9d-3b6556a48040-kube-api-access-b75jt\") pod \"payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.585004 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.584883 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc06679e-69e2-4121-9a9d-3b6556a48040-plugins-config-volume\") pod \"payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.585803 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.585775 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc06679e-69e2-4121-9a9d-3b6556a48040-plugins-config-volume\") pod \"payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.594349 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.594314 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:41.604245 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.603643 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75jt\" (UniqueName: \"kubernetes.io/projected/fc06679e-69e2-4121-9a9d-3b6556a48040-kube-api-access-b75jt\") pod \"payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.678704 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.678673 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:41.900953 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.900928 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc"] Sep 15 12:38:41.902820 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:41.902793 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b3e809_4391_48b7_845d_796a3f1d5965.slice/crio-47317c11be6201e65e55734715c5f7cb62564b1c42279d0fd29000d30534c59e WatchSource:0}: Error finding container 47317c11be6201e65e55734715c5f7cb62564b1c42279d0fd29000d30534c59e: Status 404 returned error can't find the container with id 47317c11be6201e65e55734715c5f7cb62564b1c42279d0fd29000d30534c59e Sep 15 12:38:41.960646 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.960581 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:38:41.962107 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.962080 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:38:41.990540 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:41.990513 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6"] Sep 15 12:38:41.991529 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:41.991505 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc06679e_69e2_4121_9a9d_3b6556a48040.slice/crio-bcfca9d9117f957fe55492e42c17fa42d317737c7ee715ed56a9ca83c7f470a6 WatchSource:0}: Error finding container bcfca9d9117f957fe55492e42c17fa42d317737c7ee715ed56a9ca83c7f470a6: Status 404 returned error can't find the container with id bcfca9d9117f957fe55492e42c17fa42d317737c7ee715ed56a9ca83c7f470a6 Sep 15 12:38:42.023118 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:42.023076 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" event={"ID":"fc06679e-69e2-4121-9a9d-3b6556a48040","Type":"ContainerStarted","Data":"bcfca9d9117f957fe55492e42c17fa42d317737c7ee715ed56a9ca83c7f470a6"} Sep 15 12:38:42.025417 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:42.025365 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" event={"ID":"c3b3e809-4391-48b7-845d-796a3f1d5965","Type":"ContainerStarted","Data":"81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05"} Sep 15 12:38:42.025533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:42.025441 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" event={"ID":"c3b3e809-4391-48b7-845d-796a3f1d5965","Type":"ContainerStarted","Data":"47317c11be6201e65e55734715c5f7cb62564b1c42279d0fd29000d30534c59e"} Sep 15 12:38:42.025807 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:42.025783 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:43.033025 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:43.032983 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" event={"ID":"fc06679e-69e2-4121-9a9d-3b6556a48040","Type":"ContainerStarted","Data":"f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1"} Sep 15 12:38:43.033511 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:43.033068 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:43.050675 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:43.050603 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" podStartSLOduration=2.050579677 podStartE2EDuration="2.050579677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:42.053002415 +0000 UTC m=+803.407848738" watchObservedRunningTime="2026-09-15 12:38:43.050579677 +0000 UTC m=+804.405426002" Sep 15 12:38:43.982199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:43.982168 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:38:43.982404 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:43.982361 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:38:43.998242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:43.998184 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" podStartSLOduration=2.998164046 podStartE2EDuration="2.998164046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:38:43.053137954 +0000 UTC m=+804.407984277" watchObservedRunningTime="2026-09-15 12:38:43.998164046 +0000 UTC m=+805.353010370" Sep 15 12:38:44.988820 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:44.988785 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:38:44.989241 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:44.988946 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:38:47.063624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:47.063594 2550 generic.go:356] "Generic (PLEG): container finished" podID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerID="3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c" exitCode=0 Sep 15 12:38:47.064025 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:47.063665 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" event={"ID":"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3","Type":"ContainerDied","Data":"3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c"} Sep 15 12:38:47.064086 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:47.064072 2550 scope.go:117] "RemoveContainer" containerID="3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c" Sep 15 12:38:47.065297 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:47.065271 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:38:47.065359 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:47.065333 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:38:47.065436 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:47.065372 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:38:48.070934 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:48.070901 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" event={"ID":"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3","Type":"ContainerStarted","Data":"5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11"} Sep 15 12:38:48.380922 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:48.380833 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:48.381071 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:48.381052 2550 patch_prober.go:28] interesting pod/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 15 12:38:48.381112 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:48.381092 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 15 12:38:49.380573 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:49.380536 2550 patch_prober.go:28] interesting pod/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 15 12:38:49.380975 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:49.380592 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 15 12:38:50.381247 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:50.381211 2550 patch_prober.go:28] interesting pod/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 15 12:38:50.381653 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:50.381269 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 15 12:38:51.380305 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:51.380258 2550 patch_prober.go:28] interesting pod/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 15 12:38:51.380473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:51.380324 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 15 12:38:52.380571 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:52.380534 2550 patch_prober.go:28] interesting pod/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 15 12:38:52.380947 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:52.380591 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 15 12:38:53.036593 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:53.036557 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:38:53.380846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:53.380749 2550 patch_prober.go:28] interesting pod/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Sep 15 12:38:53.380846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:53.380819 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Sep 15 12:38:54.043072 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:54.043043 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:38:54.384261 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:54.384189 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:54.384625 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:54.384423 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:54.385047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:54.385031 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:38:58.133396 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.133353 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l"] Sep 15 12:38:58.138616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.138591 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" Sep 15 12:38:58.140725 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.140706 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:38:58.140824 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.140712 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:38:58.144961 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.144936 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l"] Sep 15 12:38:58.253218 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.253179 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516a4bf3-47f3-47f1-8268-6338f27bb7ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1f5381de-bql8l\" (UID: \"516a4bf3-47f3-47f1-8268-6338f27bb7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" Sep 15 12:38:58.354216 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.354178 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516a4bf3-47f3-47f1-8268-6338f27bb7ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1f5381de-bql8l\" (UID: \"516a4bf3-47f3-47f1-8268-6338f27bb7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" Sep 15 12:38:58.354805 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.354785 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516a4bf3-47f3-47f1-8268-6338f27bb7ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1f5381de-bql8l\" (UID: \"516a4bf3-47f3-47f1-8268-6338f27bb7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" Sep 15 12:38:58.452354 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.452262 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" Sep 15 12:38:58.731784 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:58.731758 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l"] Sep 15 12:38:58.734081 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:38:58.734041 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516a4bf3_47f3_47f1_8268_6338f27bb7ae.slice/crio-925f18370d03b869b596921e09eefd466f147ccda3b0542e12b3bcac4e52995b WatchSource:0}: Error finding container 925f18370d03b869b596921e09eefd466f147ccda3b0542e12b3bcac4e52995b: Status 404 returned error can't find the container with id 925f18370d03b869b596921e09eefd466f147ccda3b0542e12b3bcac4e52995b Sep 15 12:38:59.133195 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:38:59.133158 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" event={"ID":"516a4bf3-47f3-47f1-8268-6338f27bb7ae","Type":"ContainerStarted","Data":"925f18370d03b869b596921e09eefd466f147ccda3b0542e12b3bcac4e52995b"} Sep 15 12:39:02.151135 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.151097 2550 generic.go:356] "Generic (PLEG): container finished" podID="516a4bf3-47f3-47f1-8268-6338f27bb7ae" containerID="aa87f90511a3047c73c7f702436d95f784f6d1f1a36a9f16b22942c6663086b5" exitCode=0 Sep 15 12:39:02.151135 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.151127 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" event={"ID":"516a4bf3-47f3-47f1-8268-6338f27bb7ae","Type":"ContainerDied","Data":"aa87f90511a3047c73c7f702436d95f784f6d1f1a36a9f16b22942c6663086b5"} Sep 15 12:39:02.397874 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.397840 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg"] Sep 15 12:39:02.401664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.401616 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.403884 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.403857 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2e3fef-dockercfg-qtqk5\"" Sep 15 12:39:02.404004 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.403904 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2e3fef\"" Sep 15 12:39:02.410019 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.409992 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg"] Sep 15 12:39:02.466847 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.466815 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc"] Sep 15 12:39:02.470426 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.470376 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.478087 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.478057 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc"] Sep 15 12:39:02.505696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.505656 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3a1a2d-221c-474f-b440-923914c51eef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.505966 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.505856 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s7p9\" (UniqueName: \"kubernetes.io/projected/0e3a1a2d-221c-474f-b440-923914c51eef-kube-api-access-6s7p9\") pod \"payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.607301 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.607264 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tl6n\" (UniqueName: \"kubernetes.io/projected/77304bd3-8e21-4e0f-8004-4b178d077315-kube-api-access-8tl6n\") pod \"payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.607301 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.607303 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77304bd3-8e21-4e0f-8004-4b178d077315-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.607570 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.607335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s7p9\" (UniqueName: \"kubernetes.io/projected/0e3a1a2d-221c-474f-b440-923914c51eef-kube-api-access-6s7p9\") pod \"payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.607636 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.607561 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3a1a2d-221c-474f-b440-923914c51eef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.608275 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.608255 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3a1a2d-221c-474f-b440-923914c51eef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.614707 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.614683 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s7p9\" (UniqueName: \"kubernetes.io/projected/0e3a1a2d-221c-474f-b440-923914c51eef-kube-api-access-6s7p9\") pod \"payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.708607 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.708513 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tl6n\" (UniqueName: \"kubernetes.io/projected/77304bd3-8e21-4e0f-8004-4b178d077315-kube-api-access-8tl6n\") pod \"payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.708607 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.708561 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77304bd3-8e21-4e0f-8004-4b178d077315-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.709274 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.709253 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77304bd3-8e21-4e0f-8004-4b178d077315-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.713443 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.713424 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:02.716449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.716430 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tl6n\" (UniqueName: \"kubernetes.io/projected/77304bd3-8e21-4e0f-8004-4b178d077315-kube-api-access-8tl6n\") pod \"payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.781855 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.781819 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:02.986587 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:02.986557 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg"] Sep 15 12:39:02.988452 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:39:02.988427 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3a1a2d_221c_474f_b440_923914c51eef.slice/crio-1c1637a9c78852bed050954f0bcc67039fe24b7a1b19bc3293c15cac8ba71414 WatchSource:0}: Error finding container 1c1637a9c78852bed050954f0bcc67039fe24b7a1b19bc3293c15cac8ba71414: Status 404 returned error can't find the container with id 1c1637a9c78852bed050954f0bcc67039fe24b7a1b19bc3293c15cac8ba71414 Sep 15 12:39:03.067398 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.067332 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc"] Sep 15 12:39:03.070147 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:39:03.070124 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77304bd3_8e21_4e0f_8004_4b178d077315.slice/crio-42d1ef5619186415d034dff6ec4806a730327a46d2063dc1ece0c55de8161944 WatchSource:0}: Error finding container 42d1ef5619186415d034dff6ec4806a730327a46d2063dc1ece0c55de8161944: Status 404 returned error can't find the container with id 42d1ef5619186415d034dff6ec4806a730327a46d2063dc1ece0c55de8161944 Sep 15 12:39:03.159834 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.159778 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" event={"ID":"0e3a1a2d-221c-474f-b440-923914c51eef","Type":"ContainerStarted","Data":"6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2"} Sep 15 12:39:03.160368 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.159849 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" event={"ID":"0e3a1a2d-221c-474f-b440-923914c51eef","Type":"ContainerStarted","Data":"1c1637a9c78852bed050954f0bcc67039fe24b7a1b19bc3293c15cac8ba71414"} Sep 15 12:39:03.160368 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.159882 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:03.161638 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.161603 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" event={"ID":"77304bd3-8e21-4e0f-8004-4b178d077315","Type":"ContainerStarted","Data":"e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc"} Sep 15 12:39:03.161753 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.161656 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" event={"ID":"77304bd3-8e21-4e0f-8004-4b178d077315","Type":"ContainerStarted","Data":"42d1ef5619186415d034dff6ec4806a730327a46d2063dc1ece0c55de8161944"} Sep 15 12:39:03.178732 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.178682 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" podStartSLOduration=1.178666502 podStartE2EDuration="1.178666502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:03.175920533 +0000 UTC m=+824.530766857" watchObservedRunningTime="2026-09-15 12:39:03.178666502 +0000 UTC m=+824.533512824" Sep 15 12:39:03.191304 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.191250 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" podStartSLOduration=1.191234603 podStartE2EDuration="1.191234603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:03.189893322 +0000 UTC m=+824.544739644" watchObservedRunningTime="2026-09-15 12:39:03.191234603 +0000 UTC m=+824.546080925" Sep 15 12:39:03.298479 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.298452 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" Sep 15 12:39:03.416856 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.416831 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516a4bf3-47f3-47f1-8268-6338f27bb7ae-maas-api-service-ca\") pod \"516a4bf3-47f3-47f1-8268-6338f27bb7ae\" (UID: \"516a4bf3-47f3-47f1-8268-6338f27bb7ae\") " Sep 15 12:39:03.417204 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.417179 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/516a4bf3-47f3-47f1-8268-6338f27bb7ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "516a4bf3-47f3-47f1-8268-6338f27bb7ae" (UID: "516a4bf3-47f3-47f1-8268-6338f27bb7ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:39:03.517532 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:03.517498 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/516a4bf3-47f3-47f1-8268-6338f27bb7ae-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:39:04.168766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:04.168738 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" Sep 15 12:39:04.169221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:04.168782 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l" event={"ID":"516a4bf3-47f3-47f1-8268-6338f27bb7ae","Type":"ContainerDied","Data":"925f18370d03b869b596921e09eefd466f147ccda3b0542e12b3bcac4e52995b"} Sep 15 12:39:04.169221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:04.168833 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925f18370d03b869b596921e09eefd466f147ccda3b0542e12b3bcac4e52995b" Sep 15 12:39:04.169221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:04.169215 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:08.811066 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:08.811028 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l"] Sep 15 12:39:08.814540 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:08.814514 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1f5381de-bql8l"] Sep 15 12:39:09.154321 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:09.154240 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="516a4bf3-47f3-47f1-8268-6338f27bb7ae" path="/var/lib/kubelet/pods/516a4bf3-47f3-47f1-8268-6338f27bb7ae/volumes" Sep 15 12:39:14.171732 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:14.171699 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:39:15.177545 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:15.177481 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:39:17.251129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.251090 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj"] Sep 15 12:39:17.251616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.251602 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="516a4bf3-47f3-47f1-8268-6338f27bb7ae" containerName="revoke-keys" Sep 15 12:39:17.251662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.251618 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="516a4bf3-47f3-47f1-8268-6338f27bb7ae" containerName="revoke-keys" Sep 15 12:39:17.251718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.251709 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="516a4bf3-47f3-47f1-8268-6338f27bb7ae" containerName="revoke-keys" Sep 15 12:39:17.254751 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.254734 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.256928 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.256906 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ecae662b-dockercfg-grh9d\"" Sep 15 12:39:17.257049 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.256914 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ecae662b\"" Sep 15 12:39:17.264362 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.264341 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj"] Sep 15 12:39:17.363880 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.363845 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/cb3c9a57-3517-4492-bde2-05332cabe9f9-kube-api-access-npgpl\") pod \"payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.364051 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.364007 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c9a57-3517-4492-bde2-05332cabe9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.465436 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.465357 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c9a57-3517-4492-bde2-05332cabe9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.465603 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.465461 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/cb3c9a57-3517-4492-bde2-05332cabe9f9-kube-api-access-npgpl\") pod \"payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.466071 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.466049 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c9a57-3517-4492-bde2-05332cabe9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.472744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.472716 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/cb3c9a57-3517-4492-bde2-05332cabe9f9-kube-api-access-npgpl\") pod \"payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.566578 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.566495 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:17.842831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:17.842803 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj"] Sep 15 12:39:17.844736 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:39:17.844698 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3c9a57_3517_4492_bde2_05332cabe9f9.slice/crio-efa6708f75d8108585e9f684b03b1eeef6a7dc8751c60174ca342595a7e74647 WatchSource:0}: Error finding container efa6708f75d8108585e9f684b03b1eeef6a7dc8751c60174ca342595a7e74647: Status 404 returned error can't find the container with id efa6708f75d8108585e9f684b03b1eeef6a7dc8751c60174ca342595a7e74647 Sep 15 12:39:18.246572 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:18.246533 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" event={"ID":"cb3c9a57-3517-4492-bde2-05332cabe9f9","Type":"ContainerStarted","Data":"e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924"} Sep 15 12:39:18.246749 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:18.246595 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" event={"ID":"cb3c9a57-3517-4492-bde2-05332cabe9f9","Type":"ContainerStarted","Data":"efa6708f75d8108585e9f684b03b1eeef6a7dc8751c60174ca342595a7e74647"} Sep 15 12:39:18.246749 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:18.246694 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:18.261836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:18.261784 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" podStartSLOduration=1.261767456 podStartE2EDuration="1.261767456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:18.260857767 +0000 UTC m=+839.615704093" watchObservedRunningTime="2026-09-15 12:39:18.261767456 +0000 UTC m=+839.616613778" Sep 15 12:39:29.254296 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:29.254264 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:43.202782 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.202750 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn"] Sep 15 12:39:43.210480 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.210454 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" Sep 15 12:39:43.212785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.212759 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:39:43.213048 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.213030 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:39:43.214170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.214143 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn"] Sep 15 12:39:43.317978 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.317949 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn\" (UID: \"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" Sep 15 12:39:43.418786 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.418755 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn\" (UID: \"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" Sep 15 12:39:43.419354 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.419332 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn\" (UID: \"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" Sep 15 12:39:43.523334 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.523305 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" Sep 15 12:39:43.796330 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:43.796258 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn"] Sep 15 12:39:43.799480 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:39:43.799446 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a0a3f4_1b54_4aa9_a93a_98428ba2f7a7.slice/crio-c150cdaef92e1d7d4ee52b1fa29efbbea5c363fb48fa88b4996424d207fd722c WatchSource:0}: Error finding container c150cdaef92e1d7d4ee52b1fa29efbbea5c363fb48fa88b4996424d207fd722c: Status 404 returned error can't find the container with id c150cdaef92e1d7d4ee52b1fa29efbbea5c363fb48fa88b4996424d207fd722c Sep 15 12:39:44.393449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:44.393340 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" event={"ID":"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7","Type":"ContainerStarted","Data":"2d81a85b308841bd4e3b5a564008d5cf15f8da260299a5d60496234ac6b8a77c"} Sep 15 12:39:44.393449 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:44.393411 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" event={"ID":"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7","Type":"ContainerStarted","Data":"c150cdaef92e1d7d4ee52b1fa29efbbea5c363fb48fa88b4996424d207fd722c"} Sep 15 12:39:44.409678 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:44.409632 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" podStartSLOduration=1.409615696 podStartE2EDuration="1.409615696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:39:44.407778666 +0000 UTC m=+865.762625002" watchObservedRunningTime="2026-09-15 12:39:44.409615696 +0000 UTC m=+865.764462018" Sep 15 12:39:45.400066 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:45.400030 2550 generic.go:356] "Generic (PLEG): container finished" podID="c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7" containerID="2d81a85b308841bd4e3b5a564008d5cf15f8da260299a5d60496234ac6b8a77c" exitCode=0 Sep 15 12:39:45.400477 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:45.400103 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" event={"ID":"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7","Type":"ContainerDied","Data":"2d81a85b308841bd4e3b5a564008d5cf15f8da260299a5d60496234ac6b8a77c"} Sep 15 12:39:46.558534 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:46.558509 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" Sep 15 12:39:46.648555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:46.648520 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7-maas-api-service-ca\") pod \"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7\" (UID: \"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7\") " Sep 15 12:39:46.648852 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:46.648829 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7" (UID: "c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:39:46.648922 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:46.648911 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:39:47.413912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:47.413822 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" Sep 15 12:39:47.413912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:47.413859 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn" event={"ID":"c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7","Type":"ContainerDied","Data":"c150cdaef92e1d7d4ee52b1fa29efbbea5c363fb48fa88b4996424d207fd722c"} Sep 15 12:39:47.413912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:47.413906 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c150cdaef92e1d7d4ee52b1fa29efbbea5c363fb48fa88b4996424d207fd722c" Sep 15 12:39:54.921457 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:54.921415 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj"] Sep 15 12:39:54.922035 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:54.921657 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" podUID="cb3c9a57-3517-4492-bde2-05332cabe9f9" containerName="payload-pre-processing" containerID="cri-o://e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924" gracePeriod=30 Sep 15 12:39:55.188836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.188805 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn"] Sep 15 12:39:55.191581 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.191555 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ecae662b-hpwvn"] Sep 15 12:39:55.211059 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.211020 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:55.332892 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.332852 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/cb3c9a57-3517-4492-bde2-05332cabe9f9-kube-api-access-npgpl\") pod \"cb3c9a57-3517-4492-bde2-05332cabe9f9\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " Sep 15 12:39:55.333046 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.333031 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c9a57-3517-4492-bde2-05332cabe9f9-plugins-config-volume\") pod \"cb3c9a57-3517-4492-bde2-05332cabe9f9\" (UID: \"cb3c9a57-3517-4492-bde2-05332cabe9f9\") " Sep 15 12:39:55.333478 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.333444 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3c9a57-3517-4492-bde2-05332cabe9f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb3c9a57-3517-4492-bde2-05332cabe9f9" (UID: "cb3c9a57-3517-4492-bde2-05332cabe9f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:39:55.335144 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.335119 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3c9a57-3517-4492-bde2-05332cabe9f9-kube-api-access-npgpl" (OuterVolumeSpecName: "kube-api-access-npgpl") pod "cb3c9a57-3517-4492-bde2-05332cabe9f9" (UID: "cb3c9a57-3517-4492-bde2-05332cabe9f9"). InnerVolumeSpecName "kube-api-access-npgpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:39:55.434557 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.434511 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3c9a57-3517-4492-bde2-05332cabe9f9-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:39:55.434557 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.434547 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/cb3c9a57-3517-4492-bde2-05332cabe9f9-kube-api-access-npgpl\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:39:55.458826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.458728 2550 generic.go:356] "Generic (PLEG): container finished" podID="cb3c9a57-3517-4492-bde2-05332cabe9f9" containerID="e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924" exitCode=0 Sep 15 12:39:55.458826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.458773 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" Sep 15 12:39:55.458826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.458792 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" event={"ID":"cb3c9a57-3517-4492-bde2-05332cabe9f9","Type":"ContainerDied","Data":"e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924"} Sep 15 12:39:55.459042 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.458834 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj" event={"ID":"cb3c9a57-3517-4492-bde2-05332cabe9f9","Type":"ContainerDied","Data":"efa6708f75d8108585e9f684b03b1eeef6a7dc8751c60174ca342595a7e74647"} Sep 15 12:39:55.459042 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.458857 2550 scope.go:117] "RemoveContainer" containerID="e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924" Sep 15 12:39:55.469518 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.469495 2550 scope.go:117] "RemoveContainer" containerID="e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924" Sep 15 12:39:55.469867 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:39:55.469845 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924\": container with ID starting with e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924 not found: ID does not exist" containerID="e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924" Sep 15 12:39:55.469962 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.469873 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924"} err="failed to get container status \"e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924\": rpc error: code = NotFound desc = could not find container \"e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924\": container with ID starting with e8f0eb11af9351cdb0e03b95390705a21a5c66f978cf6d397475a250afa93924 not found: ID does not exist" Sep 15 12:39:55.482705 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.482668 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj"] Sep 15 12:39:55.485689 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:55.485662 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ecae662b-7b6f5dfcd-g7ngj"] Sep 15 12:39:57.154738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:57.154698 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7" path="/var/lib/kubelet/pods/c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7/volumes" Sep 15 12:39:57.155129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:39:57.155040 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3c9a57-3517-4492-bde2-05332cabe9f9" path="/var/lib/kubelet/pods/cb3c9a57-3517-4492-bde2-05332cabe9f9/volumes" Sep 15 12:40:19.128483 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:19.128411 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:40:19.137966 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:19.137938 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:40:23.967860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.967825 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7"] Sep 15 12:40:23.968301 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.968288 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7" containerName="revoke-keys" Sep 15 12:40:23.968358 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.968303 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7" containerName="revoke-keys" Sep 15 12:40:23.968358 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.968314 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3c9a57-3517-4492-bde2-05332cabe9f9" containerName="payload-pre-processing" Sep 15 12:40:23.968358 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.968320 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3c9a57-3517-4492-bde2-05332cabe9f9" containerName="payload-pre-processing" Sep 15 12:40:23.968504 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.968406 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a0a3f4-1b54-4aa9-a93a-98428ba2f7a7" containerName="revoke-keys" Sep 15 12:40:23.968504 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.968417 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3c9a57-3517-4492-bde2-05332cabe9f9" containerName="payload-pre-processing" Sep 15 12:40:23.972727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.972711 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" Sep 15 12:40:23.975664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.975427 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:40:23.975664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.975438 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:40:23.978118 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:23.978094 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7"] Sep 15 12:40:24.113900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:24.113861 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/443d35c1-3915-4aca-aa7b-b00367fca2fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7\" (UID: \"443d35c1-3915-4aca-aa7b-b00367fca2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" Sep 15 12:40:24.215455 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:24.215406 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/443d35c1-3915-4aca-aa7b-b00367fca2fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7\" (UID: \"443d35c1-3915-4aca-aa7b-b00367fca2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" Sep 15 12:40:24.216088 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:24.216060 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/443d35c1-3915-4aca-aa7b-b00367fca2fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7\" (UID: \"443d35c1-3915-4aca-aa7b-b00367fca2fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" Sep 15 12:40:24.291182 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:24.291151 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" Sep 15 12:40:24.558910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:24.558883 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7"] Sep 15 12:40:24.560732 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:40:24.560707 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443d35c1_3915_4aca_aa7b_b00367fca2fb.slice/crio-29c1523890839c1dc81dee4adc408b4e821ff21f03e9fbae28fdec98e1079bbb WatchSource:0}: Error finding container 29c1523890839c1dc81dee4adc408b4e821ff21f03e9fbae28fdec98e1079bbb: Status 404 returned error can't find the container with id 29c1523890839c1dc81dee4adc408b4e821ff21f03e9fbae28fdec98e1079bbb Sep 15 12:40:24.621031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:24.620994 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" event={"ID":"443d35c1-3915-4aca-aa7b-b00367fca2fb","Type":"ContainerStarted","Data":"29c1523890839c1dc81dee4adc408b4e821ff21f03e9fbae28fdec98e1079bbb"} Sep 15 12:40:25.628404 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:25.628348 2550 generic.go:356] "Generic (PLEG): container finished" podID="443d35c1-3915-4aca-aa7b-b00367fca2fb" containerID="9ba652f8e0827de386b6618f242a0c05d8f9180f2ea643086ca88a5b90ec916e" exitCode=0 Sep 15 12:40:25.628873 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:25.628416 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" event={"ID":"443d35c1-3915-4aca-aa7b-b00367fca2fb","Type":"ContainerDied","Data":"9ba652f8e0827de386b6618f242a0c05d8f9180f2ea643086ca88a5b90ec916e"} Sep 15 12:40:26.777202 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:26.777172 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" Sep 15 12:40:26.841761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:26.841726 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/443d35c1-3915-4aca-aa7b-b00367fca2fb-maas-api-service-ca\") pod \"443d35c1-3915-4aca-aa7b-b00367fca2fb\" (UID: \"443d35c1-3915-4aca-aa7b-b00367fca2fb\") " Sep 15 12:40:26.842148 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:26.842119 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443d35c1-3915-4aca-aa7b-b00367fca2fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "443d35c1-3915-4aca-aa7b-b00367fca2fb" (UID: "443d35c1-3915-4aca-aa7b-b00367fca2fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:40:26.942738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:26.942649 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/443d35c1-3915-4aca-aa7b-b00367fca2fb-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:40:27.641520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:27.641488 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" Sep 15 12:40:27.641520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:27.641510 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7" event={"ID":"443d35c1-3915-4aca-aa7b-b00367fca2fb","Type":"ContainerDied","Data":"29c1523890839c1dc81dee4adc408b4e821ff21f03e9fbae28fdec98e1079bbb"} Sep 15 12:40:27.641726 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:27.641557 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29c1523890839c1dc81dee4adc408b4e821ff21f03e9fbae28fdec98e1079bbb" Sep 15 12:40:30.790327 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:30.790288 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7"] Sep 15 12:40:30.792316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:30.792293 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-81247cf5-fxjv7"] Sep 15 12:40:31.154224 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:31.154143 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="443d35c1-3915-4aca-aa7b-b00367fca2fb" path="/var/lib/kubelet/pods/443d35c1-3915-4aca-aa7b-b00367fca2fb/volumes" Sep 15 12:40:44.350394 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.350350 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b"] Sep 15 12:40:44.350844 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.350830 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="443d35c1-3915-4aca-aa7b-b00367fca2fb" containerName="revoke-keys" Sep 15 12:40:44.350888 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.350845 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="443d35c1-3915-4aca-aa7b-b00367fca2fb" containerName="revoke-keys" Sep 15 12:40:44.350927 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.350910 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="443d35c1-3915-4aca-aa7b-b00367fca2fb" containerName="revoke-keys" Sep 15 12:40:44.354338 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.354321 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" Sep 15 12:40:44.356541 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.356515 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:40:44.356650 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.356572 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:40:44.360883 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.360862 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b"] Sep 15 12:40:44.404856 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.404823 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d82b292-ae42-4383-988a-975e08b9afa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b\" (UID: \"7d82b292-ae42-4383-988a-975e08b9afa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" Sep 15 12:40:44.505826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.505790 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d82b292-ae42-4383-988a-975e08b9afa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b\" (UID: \"7d82b292-ae42-4383-988a-975e08b9afa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" Sep 15 12:40:44.506526 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.506501 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d82b292-ae42-4383-988a-975e08b9afa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b\" (UID: \"7d82b292-ae42-4383-988a-975e08b9afa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" Sep 15 12:40:44.668942 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.668851 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" Sep 15 12:40:44.937579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:44.937552 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b"] Sep 15 12:40:44.939760 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:40:44.939734 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82b292_ae42_4383_988a_975e08b9afa0.slice/crio-018c18d5a8354b48c21278bfd0f2236d64f3f4bfe5d5f3697036ebcd2747d4ae WatchSource:0}: Error finding container 018c18d5a8354b48c21278bfd0f2236d64f3f4bfe5d5f3697036ebcd2747d4ae: Status 404 returned error can't find the container with id 018c18d5a8354b48c21278bfd0f2236d64f3f4bfe5d5f3697036ebcd2747d4ae Sep 15 12:40:45.751597 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:45.751557 2550 generic.go:356] "Generic (PLEG): container finished" podID="7d82b292-ae42-4383-988a-975e08b9afa0" containerID="eb260612cd9d76d7834995d8f34fadf540ba15000f2487dde132b8f5af729ede" exitCode=0 Sep 15 12:40:45.751967 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:45.751646 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" event={"ID":"7d82b292-ae42-4383-988a-975e08b9afa0","Type":"ContainerDied","Data":"eb260612cd9d76d7834995d8f34fadf540ba15000f2487dde132b8f5af729ede"} Sep 15 12:40:45.751967 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:45.751712 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" event={"ID":"7d82b292-ae42-4383-988a-975e08b9afa0","Type":"ContainerStarted","Data":"018c18d5a8354b48c21278bfd0f2236d64f3f4bfe5d5f3697036ebcd2747d4ae"} Sep 15 12:40:46.911985 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:46.911956 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" Sep 15 12:40:46.924854 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:46.924820 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d82b292-ae42-4383-988a-975e08b9afa0-maas-api-service-ca\") pod \"7d82b292-ae42-4383-988a-975e08b9afa0\" (UID: \"7d82b292-ae42-4383-988a-975e08b9afa0\") " Sep 15 12:40:46.925199 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:46.925174 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d82b292-ae42-4383-988a-975e08b9afa0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d82b292-ae42-4383-988a-975e08b9afa0" (UID: "7d82b292-ae42-4383-988a-975e08b9afa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:40:47.026283 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:47.026186 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d82b292-ae42-4383-988a-975e08b9afa0-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:40:47.763840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:47.763806 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" event={"ID":"7d82b292-ae42-4383-988a-975e08b9afa0","Type":"ContainerDied","Data":"018c18d5a8354b48c21278bfd0f2236d64f3f4bfe5d5f3697036ebcd2747d4ae"} Sep 15 12:40:47.764008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:47.763859 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="018c18d5a8354b48c21278bfd0f2236d64f3f4bfe5d5f3697036ebcd2747d4ae" Sep 15 12:40:47.764008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:47.763817 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b" Sep 15 12:40:54.928004 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:54.927966 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6"] Sep 15 12:40:54.928640 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:54.928185 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" podUID="fc06679e-69e2-4121-9a9d-3b6556a48040" containerName="payload-processing" containerID="cri-o://f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1" gracePeriod=30 Sep 15 12:40:54.941107 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:54.941080 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc"] Sep 15 12:40:54.941342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:54.941303 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" podUID="c3b3e809-4391-48b7-845d-796a3f1d5965" containerName="payload-pre-processing" containerID="cri-o://81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05" gracePeriod=30 Sep 15 12:40:55.138805 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.138764 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b"] Sep 15 12:40:55.143686 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.143653 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bf37f4-f6t4b"] Sep 15 12:40:55.159064 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.159027 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d82b292-ae42-4383-988a-975e08b9afa0" path="/var/lib/kubelet/pods/7d82b292-ae42-4383-988a-975e08b9afa0/volumes" Sep 15 12:40:55.246514 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.246172 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:40:55.253630 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.253597 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:40:55.306082 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.306051 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbzrs\" (UniqueName: \"kubernetes.io/projected/c3b3e809-4391-48b7-845d-796a3f1d5965-kube-api-access-qbzrs\") pod \"c3b3e809-4391-48b7-845d-796a3f1d5965\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " Sep 15 12:40:55.306264 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.306133 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b3e809-4391-48b7-845d-796a3f1d5965-plugins-config-volume\") pod \"c3b3e809-4391-48b7-845d-796a3f1d5965\" (UID: \"c3b3e809-4391-48b7-845d-796a3f1d5965\") " Sep 15 12:40:55.306264 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.306168 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc06679e-69e2-4121-9a9d-3b6556a48040-plugins-config-volume\") pod \"fc06679e-69e2-4121-9a9d-3b6556a48040\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " Sep 15 12:40:55.306264 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.306195 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b75jt\" (UniqueName: \"kubernetes.io/projected/fc06679e-69e2-4121-9a9d-3b6556a48040-kube-api-access-b75jt\") pod \"fc06679e-69e2-4121-9a9d-3b6556a48040\" (UID: \"fc06679e-69e2-4121-9a9d-3b6556a48040\") " Sep 15 12:40:55.306582 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.306554 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc06679e-69e2-4121-9a9d-3b6556a48040-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc06679e-69e2-4121-9a9d-3b6556a48040" (UID: "fc06679e-69e2-4121-9a9d-3b6556a48040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:40:55.306652 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.306554 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b3e809-4391-48b7-845d-796a3f1d5965-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3b3e809-4391-48b7-845d-796a3f1d5965" (UID: "c3b3e809-4391-48b7-845d-796a3f1d5965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:40:55.308187 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.308167 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b3e809-4391-48b7-845d-796a3f1d5965-kube-api-access-qbzrs" (OuterVolumeSpecName: "kube-api-access-qbzrs") pod "c3b3e809-4391-48b7-845d-796a3f1d5965" (UID: "c3b3e809-4391-48b7-845d-796a3f1d5965"). InnerVolumeSpecName "kube-api-access-qbzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:40:55.308273 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.308247 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc06679e-69e2-4121-9a9d-3b6556a48040-kube-api-access-b75jt" (OuterVolumeSpecName: "kube-api-access-b75jt") pod "fc06679e-69e2-4121-9a9d-3b6556a48040" (UID: "fc06679e-69e2-4121-9a9d-3b6556a48040"). InnerVolumeSpecName "kube-api-access-b75jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:40:55.407041 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.406999 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc06679e-69e2-4121-9a9d-3b6556a48040-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:40:55.407041 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.407035 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b75jt\" (UniqueName: \"kubernetes.io/projected/fc06679e-69e2-4121-9a9d-3b6556a48040-kube-api-access-b75jt\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:40:55.407041 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.407045 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbzrs\" (UniqueName: \"kubernetes.io/projected/c3b3e809-4391-48b7-845d-796a3f1d5965-kube-api-access-qbzrs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:40:55.407279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.407056 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b3e809-4391-48b7-845d-796a3f1d5965-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:40:55.807846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.807814 2550 generic.go:356] "Generic (PLEG): container finished" podID="fc06679e-69e2-4121-9a9d-3b6556a48040" containerID="f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1" exitCode=0 Sep 15 12:40:55.808027 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.807862 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" Sep 15 12:40:55.808027 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.807890 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" event={"ID":"fc06679e-69e2-4121-9a9d-3b6556a48040","Type":"ContainerDied","Data":"f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1"} Sep 15 12:40:55.808027 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.807935 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6" event={"ID":"fc06679e-69e2-4121-9a9d-3b6556a48040","Type":"ContainerDied","Data":"bcfca9d9117f957fe55492e42c17fa42d317737c7ee715ed56a9ca83c7f470a6"} Sep 15 12:40:55.808027 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.807967 2550 scope.go:117] "RemoveContainer" containerID="f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1" Sep 15 12:40:55.809478 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.809448 2550 generic.go:356] "Generic (PLEG): container finished" podID="c3b3e809-4391-48b7-845d-796a3f1d5965" containerID="81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05" exitCode=0 Sep 15 12:40:55.809624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.809510 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" Sep 15 12:40:55.809624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.809540 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" event={"ID":"c3b3e809-4391-48b7-845d-796a3f1d5965","Type":"ContainerDied","Data":"81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05"} Sep 15 12:40:55.809624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.809588 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc" event={"ID":"c3b3e809-4391-48b7-845d-796a3f1d5965","Type":"ContainerDied","Data":"47317c11be6201e65e55734715c5f7cb62564b1c42279d0fd29000d30534c59e"} Sep 15 12:40:55.819662 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.819645 2550 scope.go:117] "RemoveContainer" containerID="f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1" Sep 15 12:40:55.819989 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:40:55.819960 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1\": container with ID starting with f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1 not found: ID does not exist" containerID="f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1" Sep 15 12:40:55.820092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.819997 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1"} err="failed to get container status \"f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1\": rpc error: code = NotFound desc = could not find container \"f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1\": container with ID starting with f56aa9d7350f75d9564b586835a2fb794c6f54319334319b3ce1f8af5afdb2e1 not found: ID does not exist" Sep 15 12:40:55.820092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.820021 2550 scope.go:117] "RemoveContainer" containerID="81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05" Sep 15 12:40:55.828365 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.828351 2550 scope.go:117] "RemoveContainer" containerID="81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05" Sep 15 12:40:55.828640 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:40:55.828623 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05\": container with ID starting with 81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05 not found: ID does not exist" containerID="81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05" Sep 15 12:40:55.828677 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.828648 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05"} err="failed to get container status \"81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05\": rpc error: code = NotFound desc = could not find container \"81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05\": container with ID starting with 81fac9bae39916f1d82eaaeb9f2e9057c0b987e3c7fd730fc87e2735fd482f05 not found: ID does not exist" Sep 15 12:40:55.834346 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.834322 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc"] Sep 15 12:40:55.838753 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.838731 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bf37f4-7897cc6f74-fmjmc"] Sep 15 12:40:55.849674 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.849651 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6"] Sep 15 12:40:55.851578 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:55.851559 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bf37f4-85d98f459c-6kks6"] Sep 15 12:40:57.154109 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:57.154075 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3b3e809-4391-48b7-845d-796a3f1d5965" path="/var/lib/kubelet/pods/c3b3e809-4391-48b7-845d-796a3f1d5965/volumes" Sep 15 12:40:57.154492 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:40:57.154455 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc06679e-69e2-4121-9a9d-3b6556a48040" path="/var/lib/kubelet/pods/fc06679e-69e2-4121-9a9d-3b6556a48040/volumes" Sep 15 12:41:00.332394 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332354 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww"] Sep 15 12:41:00.332819 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332807 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc06679e-69e2-4121-9a9d-3b6556a48040" containerName="payload-processing" Sep 15 12:41:00.332868 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332821 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc06679e-69e2-4121-9a9d-3b6556a48040" containerName="payload-processing" Sep 15 12:41:00.332868 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332839 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b3e809-4391-48b7-845d-796a3f1d5965" containerName="payload-pre-processing" Sep 15 12:41:00.332868 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332846 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b3e809-4391-48b7-845d-796a3f1d5965" containerName="payload-pre-processing" Sep 15 12:41:00.332868 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332858 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d82b292-ae42-4383-988a-975e08b9afa0" containerName="revoke-keys" Sep 15 12:41:00.332868 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332864 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d82b292-ae42-4383-988a-975e08b9afa0" containerName="revoke-keys" Sep 15 12:41:00.333021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332950 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b3e809-4391-48b7-845d-796a3f1d5965" containerName="payload-pre-processing" Sep 15 12:41:00.333021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332960 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc06679e-69e2-4121-9a9d-3b6556a48040" containerName="payload-processing" Sep 15 12:41:00.333021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.332967 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d82b292-ae42-4383-988a-975e08b9afa0" containerName="revoke-keys" Sep 15 12:41:00.337618 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.337598 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" Sep 15 12:41:00.339977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.339947 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:41:00.340096 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.340032 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:41:00.344564 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.344504 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww"] Sep 15 12:41:00.457132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.457092 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b921fbd-7e73-4658-a516-161005f5289d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww\" (UID: \"9b921fbd-7e73-4658-a516-161005f5289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" Sep 15 12:41:00.557844 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.557808 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b921fbd-7e73-4658-a516-161005f5289d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww\" (UID: \"9b921fbd-7e73-4658-a516-161005f5289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" Sep 15 12:41:00.558430 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.558412 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b921fbd-7e73-4658-a516-161005f5289d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww\" (UID: \"9b921fbd-7e73-4658-a516-161005f5289d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" Sep 15 12:41:00.650894 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.650808 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" Sep 15 12:41:00.919681 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:00.919654 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww"] Sep 15 12:41:00.921235 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:41:00.921207 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b921fbd_7e73_4658_a516_161005f5289d.slice/crio-54ecfce16fa9c8d80e213fd6d3cb0b8b3bf1de0da004115bfe9afbc55ecfe4ea WatchSource:0}: Error finding container 54ecfce16fa9c8d80e213fd6d3cb0b8b3bf1de0da004115bfe9afbc55ecfe4ea: Status 404 returned error can't find the container with id 54ecfce16fa9c8d80e213fd6d3cb0b8b3bf1de0da004115bfe9afbc55ecfe4ea Sep 15 12:41:01.852455 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:01.852409 2550 generic.go:356] "Generic (PLEG): container finished" podID="9b921fbd-7e73-4658-a516-161005f5289d" containerID="1515c50a4c074a5f5a43bbd2264e0dc8ca3280fe501e50d9dadfd53dc303e9aa" exitCode=0 Sep 15 12:41:01.852878 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:01.852476 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" event={"ID":"9b921fbd-7e73-4658-a516-161005f5289d","Type":"ContainerDied","Data":"1515c50a4c074a5f5a43bbd2264e0dc8ca3280fe501e50d9dadfd53dc303e9aa"} Sep 15 12:41:01.852878 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:01.852576 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" event={"ID":"9b921fbd-7e73-4658-a516-161005f5289d","Type":"ContainerStarted","Data":"54ecfce16fa9c8d80e213fd6d3cb0b8b3bf1de0da004115bfe9afbc55ecfe4ea"} Sep 15 12:41:02.998605 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:02.998580 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" Sep 15 12:41:03.083956 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:03.083911 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b921fbd-7e73-4658-a516-161005f5289d-maas-api-service-ca\") pod \"9b921fbd-7e73-4658-a516-161005f5289d\" (UID: \"9b921fbd-7e73-4658-a516-161005f5289d\") " Sep 15 12:41:03.084282 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:03.084255 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b921fbd-7e73-4658-a516-161005f5289d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b921fbd-7e73-4658-a516-161005f5289d" (UID: "9b921fbd-7e73-4658-a516-161005f5289d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:41:03.185702 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:03.185665 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b921fbd-7e73-4658-a516-161005f5289d-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:41:03.865173 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:03.865148 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" Sep 15 12:41:03.865342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:03.865141 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww" event={"ID":"9b921fbd-7e73-4658-a516-161005f5289d","Type":"ContainerDied","Data":"54ecfce16fa9c8d80e213fd6d3cb0b8b3bf1de0da004115bfe9afbc55ecfe4ea"} Sep 15 12:41:03.865342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:03.865271 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54ecfce16fa9c8d80e213fd6d3cb0b8b3bf1de0da004115bfe9afbc55ecfe4ea" Sep 15 12:41:07.879718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:07.879682 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww"] Sep 15 12:41:07.883134 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:07.883107 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1459b7a2-4vlww"] Sep 15 12:41:09.153899 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:41:09.153864 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b921fbd-7e73-4658-a516-161005f5289d" path="/var/lib/kubelet/pods/9b921fbd-7e73-4658-a516-161005f5289d/volumes" Sep 15 12:42:42.703022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.702987 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm"] Sep 15 12:42:42.703867 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.703845 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b921fbd-7e73-4658-a516-161005f5289d" containerName="revoke-keys" Sep 15 12:42:42.703959 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.703870 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b921fbd-7e73-4658-a516-161005f5289d" containerName="revoke-keys" Sep 15 12:42:42.704029 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.704015 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b921fbd-7e73-4658-a516-161005f5289d" containerName="revoke-keys" Sep 15 12:42:42.707308 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.707288 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.709479 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.709459 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b7e2d651-openshift-default-dockercfg-z57lb\"" Sep 15 12:42:42.719633 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.719605 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm"] Sep 15 12:42:42.776948 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.776913 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-envoy\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777141 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.776967 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-certs\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777141 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.776988 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-socket\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777141 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.777083 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2qnv\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-kube-api-access-d2qnv\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777141 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.777132 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-token\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777370 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.777172 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-podinfo\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777370 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.777213 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istiod-ca-cert\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777370 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.777256 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-credential-socket\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.777370 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.777330 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-data\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.878766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878729 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-certs\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.878766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878769 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-socket\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878794 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2qnv\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-kube-api-access-d2qnv\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878815 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-token\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878838 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-podinfo\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878863 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istiod-ca-cert\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878897 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-credential-socket\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878955 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-data\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.878992 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-envoy\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879435 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.879248 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-certs\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879435 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.879305 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-socket\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879556 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.879458 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-credential-socket\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879632 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.879613 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-data\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.879691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.879678 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istiod-ca-cert\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.881262 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.881240 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-envoy\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.881490 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.881472 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-podinfo\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.887981 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.887927 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-token\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:42.888554 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:42.888532 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2qnv\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-kube-api-access-d2qnv\") pod \"e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:43.019793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:43.019747 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:43.393144 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:42:43.393117 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0ceab7_b8ed_4da0_b2a5_3c65efb146c8.slice/crio-20c98defd8cdd34aba394d9c499ca3532d1ddd94594023689ad9230ecb9dc2c8 WatchSource:0}: Error finding container 20c98defd8cdd34aba394d9c499ca3532d1ddd94594023689ad9230ecb9dc2c8: Status 404 returned error can't find the container with id 20c98defd8cdd34aba394d9c499ca3532d1ddd94594023689ad9230ecb9dc2c8 Sep 15 12:42:43.393840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:43.393815 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm"] Sep 15 12:42:43.398955 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:43.398919 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:42:43.399039 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:43.399006 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:42:43.399078 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:43.399047 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:42:43.429742 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:43.429709 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" event={"ID":"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8","Type":"ContainerStarted","Data":"20c98defd8cdd34aba394d9c499ca3532d1ddd94594023689ad9230ecb9dc2c8"} Sep 15 12:42:44.436466 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:44.436425 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" event={"ID":"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8","Type":"ContainerStarted","Data":"c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759"} Sep 15 12:42:44.459207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:44.459153 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podStartSLOduration=2.459137848 podStartE2EDuration="2.459137848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:42:44.45757937 +0000 UTC m=+1045.812425718" watchObservedRunningTime="2026-09-15 12:42:44.459137848 +0000 UTC m=+1045.813984171" Sep 15 12:42:45.020586 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:45.020552 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:45.025901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:45.025878 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:45.442339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:45.442259 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:45.443285 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:45.443265 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:42:52.818129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.818088 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs"] Sep 15 12:42:52.821957 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.821936 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:52.824277 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.824251 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b7e2d651\"" Sep 15 12:42:52.824439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.824252 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b7e2d651-dockercfg-jjbv2\"" Sep 15 12:42:52.832762 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.832734 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs"] Sep 15 12:42:52.853796 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.853763 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj"] Sep 15 12:42:52.857360 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.857336 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:52.865825 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.865798 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj"] Sep 15 12:42:52.880009 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.879973 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c858x\" (UniqueName: \"kubernetes.io/projected/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-kube-api-access-c858x\") pod \"payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:52.880194 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.880051 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:52.981035 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.980997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:52.981244 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.981082 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqb5\" (UniqueName: \"kubernetes.io/projected/da7a7ce0-503c-4ddf-bb0c-82063c84417c-kube-api-access-hkqb5\") pod \"payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:52.981244 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.981129 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7a7ce0-503c-4ddf-bb0c-82063c84417c-plugins-config-volume\") pod \"payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:52.981244 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.981152 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c858x\" (UniqueName: \"kubernetes.io/projected/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-kube-api-access-c858x\") pod \"payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:52.981814 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.981785 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:52.989102 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:52.989071 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c858x\" (UniqueName: \"kubernetes.io/projected/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-kube-api-access-c858x\") pod \"payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:53.082008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.081904 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7a7ce0-503c-4ddf-bb0c-82063c84417c-plugins-config-volume\") pod \"payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:53.082190 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.082081 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqb5\" (UniqueName: \"kubernetes.io/projected/da7a7ce0-503c-4ddf-bb0c-82063c84417c-kube-api-access-hkqb5\") pod \"payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:53.082773 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.082751 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7a7ce0-503c-4ddf-bb0c-82063c84417c-plugins-config-volume\") pod \"payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:53.089878 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.089852 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqb5\" (UniqueName: \"kubernetes.io/projected/da7a7ce0-503c-4ddf-bb0c-82063c84417c-kube-api-access-hkqb5\") pod \"payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:53.134412 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.134341 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:53.168966 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.168934 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:53.408281 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.408258 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs"] Sep 15 12:42:53.409869 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:42:53.409847 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d30b9d2_8953_450e_b8ea_ff57f2a7e49f.slice/crio-c24a31af85ef6b7deb1c09a2e0a109e510f8c64fa2ea1a701a21d44e2e502c9a WatchSource:0}: Error finding container c24a31af85ef6b7deb1c09a2e0a109e510f8c64fa2ea1a701a21d44e2e502c9a: Status 404 returned error can't find the container with id c24a31af85ef6b7deb1c09a2e0a109e510f8c64fa2ea1a701a21d44e2e502c9a Sep 15 12:42:53.450393 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.450352 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj"] Sep 15 12:42:53.451688 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:42:53.451665 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7a7ce0_503c_4ddf_bb0c_82063c84417c.slice/crio-5a3a62de3a36280e3b6a40d6c00342c8f463046c93b745e6f3513d3829b33774 WatchSource:0}: Error finding container 5a3a62de3a36280e3b6a40d6c00342c8f463046c93b745e6f3513d3829b33774: Status 404 returned error can't find the container with id 5a3a62de3a36280e3b6a40d6c00342c8f463046c93b745e6f3513d3829b33774 Sep 15 12:42:53.497070 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.497043 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" event={"ID":"da7a7ce0-503c-4ddf-bb0c-82063c84417c","Type":"ContainerStarted","Data":"5a3a62de3a36280e3b6a40d6c00342c8f463046c93b745e6f3513d3829b33774"} Sep 15 12:42:53.499324 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.499293 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" event={"ID":"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f","Type":"ContainerStarted","Data":"f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b"} Sep 15 12:42:53.499476 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.499350 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" event={"ID":"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f","Type":"ContainerStarted","Data":"c24a31af85ef6b7deb1c09a2e0a109e510f8c64fa2ea1a701a21d44e2e502c9a"} Sep 15 12:42:53.499539 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.499503 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:42:53.515112 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:53.515058 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" podStartSLOduration=1.515039496 podStartE2EDuration="1.515039496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:42:53.514225026 +0000 UTC m=+1054.869071351" watchObservedRunningTime="2026-09-15 12:42:53.515039496 +0000 UTC m=+1054.869885818" Sep 15 12:42:54.507314 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:54.507280 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" event={"ID":"da7a7ce0-503c-4ddf-bb0c-82063c84417c","Type":"ContainerStarted","Data":"e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d"} Sep 15 12:42:54.507768 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:54.507508 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:42:54.524371 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:42:54.524326 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" podStartSLOduration=2.524311135 podStartE2EDuration="2.524311135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:42:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:42:54.522444977 +0000 UTC m=+1055.877291301" watchObservedRunningTime="2026-09-15 12:42:54.524311135 +0000 UTC m=+1055.879157458" Sep 15 12:43:04.509452 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:04.509416 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:43:05.515069 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:05.515033 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:43:14.628911 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:14.628871 2550 generic.go:356] "Generic (PLEG): container finished" podID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerID="c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759" exitCode=0 Sep 15 12:43:14.629316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:14.628916 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" event={"ID":"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8","Type":"ContainerDied","Data":"c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759"} Sep 15 12:43:14.629316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:14.629294 2550 scope.go:117] "RemoveContainer" containerID="c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759" Sep 15 12:43:14.630491 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:14.630458 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:43:14.630562 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:14.630530 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:43:14.630600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:14.630560 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:43:15.442251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:15.442199 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:43:15.638673 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:15.638624 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" event={"ID":"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8","Type":"ContainerStarted","Data":"9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6"} Sep 15 12:43:16.020140 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:16.020102 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:43:16.021455 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:16.021427 2550 patch_prober.go:28] interesting pod/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 12:43:16.021564 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:16.021489 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 12:43:17.020912 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:17.020866 2550 patch_prober.go:28] interesting pod/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 12:43:17.021301 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:17.020926 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 12:43:18.020574 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:18.020540 2550 patch_prober.go:28] interesting pod/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 12:43:18.020750 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:18.020593 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 12:43:19.020525 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:19.020487 2550 patch_prober.go:28] interesting pod/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 12:43:19.020889 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:19.020577 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 12:43:20.020888 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:20.020855 2550 patch_prober.go:28] interesting pod/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 15 12:43:20.021255 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:20.020919 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 15 12:43:21.025783 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:21.025749 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:43:21.026195 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:21.026168 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:43:21.027045 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:21.027024 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:43:24.598278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.598233 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d"] Sep 15 12:43:24.602276 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.602253 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" Sep 15 12:43:24.607253 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.607217 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:43:24.608528 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.608497 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:43:24.613303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.613276 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d"] Sep 15 12:43:24.715300 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.715270 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d216-c268-4929-b6aa-92f00198c2f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f955e-5km7d\" (UID: \"3162d216-c268-4929-b6aa-92f00198c2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" Sep 15 12:43:24.816688 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.816651 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d216-c268-4929-b6aa-92f00198c2f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f955e-5km7d\" (UID: \"3162d216-c268-4929-b6aa-92f00198c2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" Sep 15 12:43:24.817235 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.817214 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d216-c268-4929-b6aa-92f00198c2f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0f955e-5km7d\" (UID: \"3162d216-c268-4929-b6aa-92f00198c2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" Sep 15 12:43:24.924624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:24.924529 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" Sep 15 12:43:25.193898 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:25.193873 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d"] Sep 15 12:43:25.195881 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:43:25.195855 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3162d216_c268_4929_b6aa_92f00198c2f5.slice/crio-a959d6a0d2a8f24322536d45b70b644b3045aa686bf572e343c53cdf633c3ffe WatchSource:0}: Error finding container a959d6a0d2a8f24322536d45b70b644b3045aa686bf572e343c53cdf633c3ffe: Status 404 returned error can't find the container with id a959d6a0d2a8f24322536d45b70b644b3045aa686bf572e343c53cdf633c3ffe Sep 15 12:43:25.708019 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:25.707930 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" event={"ID":"3162d216-c268-4929-b6aa-92f00198c2f5","Type":"ContainerStarted","Data":"62884075e3312e671631a1edc6c50b3969a5c09287b5e2281a62b22df0d40828"} Sep 15 12:43:25.708019 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:25.707982 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" event={"ID":"3162d216-c268-4929-b6aa-92f00198c2f5","Type":"ContainerStarted","Data":"a959d6a0d2a8f24322536d45b70b644b3045aa686bf572e343c53cdf633c3ffe"} Sep 15 12:43:25.728070 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:25.728020 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" podStartSLOduration=1.728000787 podStartE2EDuration="1.728000787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:43:25.725046836 +0000 UTC m=+1087.079893159" watchObservedRunningTime="2026-09-15 12:43:25.728000787 +0000 UTC m=+1087.082847109" Sep 15 12:43:26.013897 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.013862 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c"] Sep 15 12:43:26.017552 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.017536 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" Sep 15 12:43:26.025701 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.025188 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c"] Sep 15 12:43:26.130559 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.130519 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0481b781-9ee6-4890-b6ab-067e6a8e2ba3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c\" (UID: \"0481b781-9ee6-4890-b6ab-067e6a8e2ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" Sep 15 12:43:26.231145 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.231105 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0481b781-9ee6-4890-b6ab-067e6a8e2ba3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c\" (UID: \"0481b781-9ee6-4890-b6ab-067e6a8e2ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" Sep 15 12:43:26.231777 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.231752 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0481b781-9ee6-4890-b6ab-067e6a8e2ba3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c\" (UID: \"0481b781-9ee6-4890-b6ab-067e6a8e2ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" Sep 15 12:43:26.329405 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.329313 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" Sep 15 12:43:26.602655 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.602630 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c"] Sep 15 12:43:26.604302 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:43:26.604275 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0481b781_9ee6_4890_b6ab_067e6a8e2ba3.slice/crio-c5848279e4980fd5a4db35eb07bf15bd8e018f920861b38307536678a4cd6ed3 WatchSource:0}: Error finding container c5848279e4980fd5a4db35eb07bf15bd8e018f920861b38307536678a4cd6ed3: Status 404 returned error can't find the container with id c5848279e4980fd5a4db35eb07bf15bd8e018f920861b38307536678a4cd6ed3 Sep 15 12:43:26.716221 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.716186 2550 generic.go:356] "Generic (PLEG): container finished" podID="3162d216-c268-4929-b6aa-92f00198c2f5" containerID="62884075e3312e671631a1edc6c50b3969a5c09287b5e2281a62b22df0d40828" exitCode=0 Sep 15 12:43:26.716636 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.716281 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" event={"ID":"3162d216-c268-4929-b6aa-92f00198c2f5","Type":"ContainerDied","Data":"62884075e3312e671631a1edc6c50b3969a5c09287b5e2281a62b22df0d40828"} Sep 15 12:43:26.717513 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:26.717492 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" event={"ID":"0481b781-9ee6-4890-b6ab-067e6a8e2ba3","Type":"ContainerStarted","Data":"c5848279e4980fd5a4db35eb07bf15bd8e018f920861b38307536678a4cd6ed3"} Sep 15 12:43:27.724250 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:27.724164 2550 generic.go:356] "Generic (PLEG): container finished" podID="0481b781-9ee6-4890-b6ab-067e6a8e2ba3" containerID="48bef63ddf38b10e89def950b78961a8397cf4939cd4e3f28e277247505e5df7" exitCode=0 Sep 15 12:43:27.724690 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:27.724245 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" event={"ID":"0481b781-9ee6-4890-b6ab-067e6a8e2ba3","Type":"ContainerDied","Data":"48bef63ddf38b10e89def950b78961a8397cf4939cd4e3f28e277247505e5df7"} Sep 15 12:43:27.862770 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:27.862744 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" Sep 15 12:43:27.947625 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:27.947593 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d216-c268-4929-b6aa-92f00198c2f5-maas-api-service-ca\") pod \"3162d216-c268-4929-b6aa-92f00198c2f5\" (UID: \"3162d216-c268-4929-b6aa-92f00198c2f5\") " Sep 15 12:43:27.947978 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:27.947951 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3162d216-c268-4929-b6aa-92f00198c2f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3162d216-c268-4929-b6aa-92f00198c2f5" (UID: "3162d216-c268-4929-b6aa-92f00198c2f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:43:28.051162 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:28.049065 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d216-c268-4929-b6aa-92f00198c2f5-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:28.730948 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:28.730918 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" Sep 15 12:43:28.731434 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:28.730962 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d" event={"ID":"3162d216-c268-4929-b6aa-92f00198c2f5","Type":"ContainerDied","Data":"a959d6a0d2a8f24322536d45b70b644b3045aa686bf572e343c53cdf633c3ffe"} Sep 15 12:43:28.731434 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:28.731022 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a959d6a0d2a8f24322536d45b70b644b3045aa686bf572e343c53cdf633c3ffe" Sep 15 12:43:28.877731 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:28.877705 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" Sep 15 12:43:28.957590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:28.957558 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0481b781-9ee6-4890-b6ab-067e6a8e2ba3-maas-api-service-ca\") pod \"0481b781-9ee6-4890-b6ab-067e6a8e2ba3\" (UID: \"0481b781-9ee6-4890-b6ab-067e6a8e2ba3\") " Sep 15 12:43:28.957967 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:28.957948 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0481b781-9ee6-4890-b6ab-067e6a8e2ba3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0481b781-9ee6-4890-b6ab-067e6a8e2ba3" (UID: "0481b781-9ee6-4890-b6ab-067e6a8e2ba3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:43:29.058466 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:29.058436 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0481b781-9ee6-4890-b6ab-067e6a8e2ba3-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:29.737283 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:29.737249 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" Sep 15 12:43:29.737283 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:29.737266 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c" event={"ID":"0481b781-9ee6-4890-b6ab-067e6a8e2ba3","Type":"ContainerDied","Data":"c5848279e4980fd5a4db35eb07bf15bd8e018f920861b38307536678a4cd6ed3"} Sep 15 12:43:29.737727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:29.737338 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5848279e4980fd5a4db35eb07bf15bd8e018f920861b38307536678a4cd6ed3" Sep 15 12:43:35.505736 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:35.505696 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d"] Sep 15 12:43:35.508571 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:35.508549 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0f955e-5km7d"] Sep 15 12:43:37.154553 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:37.154521 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3162d216-c268-4929-b6aa-92f00198c2f5" path="/var/lib/kubelet/pods/3162d216-c268-4929-b6aa-92f00198c2f5/volumes" Sep 15 12:43:39.502568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.502533 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj"] Sep 15 12:43:39.503371 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.503333 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" podUID="da7a7ce0-503c-4ddf-bb0c-82063c84417c" containerName="payload-processing" containerID="cri-o://e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d" gracePeriod=30 Sep 15 12:43:39.511518 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.511491 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs"] Sep 15 12:43:39.511747 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.511721 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" podUID="9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" containerName="payload-pre-processing" containerID="cri-o://f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b" gracePeriod=30 Sep 15 12:43:39.674354 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.674286 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c"] Sep 15 12:43:39.677573 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.677543 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b7e2d651-f7z9c"] Sep 15 12:43:39.765902 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.765840 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:43:39.799622 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.799594 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:43:39.801714 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.801687 2550 generic.go:356] "Generic (PLEG): container finished" podID="9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" containerID="f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b" exitCode=0 Sep 15 12:43:39.801834 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.801727 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" Sep 15 12:43:39.801834 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.801770 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" event={"ID":"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f","Type":"ContainerDied","Data":"f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b"} Sep 15 12:43:39.801962 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.801852 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs" event={"ID":"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f","Type":"ContainerDied","Data":"c24a31af85ef6b7deb1c09a2e0a109e510f8c64fa2ea1a701a21d44e2e502c9a"} Sep 15 12:43:39.801962 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.801868 2550 scope.go:117] "RemoveContainer" containerID="f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b" Sep 15 12:43:39.803293 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.803272 2550 generic.go:356] "Generic (PLEG): container finished" podID="da7a7ce0-503c-4ddf-bb0c-82063c84417c" containerID="e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d" exitCode=0 Sep 15 12:43:39.803419 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.803305 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" Sep 15 12:43:39.803419 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.803312 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" event={"ID":"da7a7ce0-503c-4ddf-bb0c-82063c84417c","Type":"ContainerDied","Data":"e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d"} Sep 15 12:43:39.803419 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.803347 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj" event={"ID":"da7a7ce0-503c-4ddf-bb0c-82063c84417c","Type":"ContainerDied","Data":"5a3a62de3a36280e3b6a40d6c00342c8f463046c93b745e6f3513d3829b33774"} Sep 15 12:43:39.812525 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.812507 2550 scope.go:117] "RemoveContainer" containerID="f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b" Sep 15 12:43:39.812813 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:43:39.812795 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b\": container with ID starting with f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b not found: ID does not exist" containerID="f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b" Sep 15 12:43:39.812869 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.812821 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b"} err="failed to get container status \"f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b\": rpc error: code = NotFound desc = could not find container \"f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b\": container with ID starting with f0ef2113c79756ad78141217be3e3e41672d7a740101edcdc87dcb99d862c34b not found: ID does not exist" Sep 15 12:43:39.812869 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.812840 2550 scope.go:117] "RemoveContainer" containerID="e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d" Sep 15 12:43:39.822108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.822084 2550 scope.go:117] "RemoveContainer" containerID="e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d" Sep 15 12:43:39.822407 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:43:39.822373 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d\": container with ID starting with e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d not found: ID does not exist" containerID="e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d" Sep 15 12:43:39.822492 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.822413 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d"} err="failed to get container status \"e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d\": rpc error: code = NotFound desc = could not find container \"e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d\": container with ID starting with e22391dbb5b9721c7085ba836576fbafd7f18e3bd51baf16215ca785e371029d not found: ID does not exist" Sep 15 12:43:39.872116 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.872083 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7a7ce0-503c-4ddf-bb0c-82063c84417c-plugins-config-volume\") pod \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " Sep 15 12:43:39.872285 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.872233 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkqb5\" (UniqueName: \"kubernetes.io/projected/da7a7ce0-503c-4ddf-bb0c-82063c84417c-kube-api-access-hkqb5\") pod \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\" (UID: \"da7a7ce0-503c-4ddf-bb0c-82063c84417c\") " Sep 15 12:43:39.872538 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.872513 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7a7ce0-503c-4ddf-bb0c-82063c84417c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da7a7ce0-503c-4ddf-bb0c-82063c84417c" (UID: "da7a7ce0-503c-4ddf-bb0c-82063c84417c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:43:39.874459 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.874434 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7a7ce0-503c-4ddf-bb0c-82063c84417c-kube-api-access-hkqb5" (OuterVolumeSpecName: "kube-api-access-hkqb5") pod "da7a7ce0-503c-4ddf-bb0c-82063c84417c" (UID: "da7a7ce0-503c-4ddf-bb0c-82063c84417c"). InnerVolumeSpecName "kube-api-access-hkqb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:43:39.973732 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.973684 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-plugins-config-volume\") pod \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " Sep 15 12:43:39.973926 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.973756 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c858x\" (UniqueName: \"kubernetes.io/projected/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-kube-api-access-c858x\") pod \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\" (UID: \"9d30b9d2-8953-450e-b8ea-ff57f2a7e49f\") " Sep 15 12:43:39.974094 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.974074 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkqb5\" (UniqueName: \"kubernetes.io/projected/da7a7ce0-503c-4ddf-bb0c-82063c84417c-kube-api-access-hkqb5\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:39.974171 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.974099 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da7a7ce0-503c-4ddf-bb0c-82063c84417c-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:39.974171 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.974118 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" (UID: "9d30b9d2-8953-450e-b8ea-ff57f2a7e49f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:43:39.975861 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:39.975837 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-kube-api-access-c858x" (OuterVolumeSpecName: "kube-api-access-c858x") pod "9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" (UID: "9d30b9d2-8953-450e-b8ea-ff57f2a7e49f"). InnerVolumeSpecName "kube-api-access-c858x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:43:40.075503 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:40.075405 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c858x\" (UniqueName: \"kubernetes.io/projected/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-kube-api-access-c858x\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:40.075503 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:40.075439 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:40.124015 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:40.123990 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs"] Sep 15 12:43:40.127815 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:40.127790 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b7e2d651-55c574bff8-7k7cs"] Sep 15 12:43:40.136799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:40.136775 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj"] Sep 15 12:43:40.139561 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:40.139540 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b7e2d651-5d4d95447c-g76pj"] Sep 15 12:43:41.154544 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:41.154506 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0481b781-9ee6-4890-b6ab-067e6a8e2ba3" path="/var/lib/kubelet/pods/0481b781-9ee6-4890-b6ab-067e6a8e2ba3/volumes" Sep 15 12:43:41.154937 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:41.154900 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" path="/var/lib/kubelet/pods/9d30b9d2-8953-450e-b8ea-ff57f2a7e49f/volumes" Sep 15 12:43:41.155249 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:41.155234 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da7a7ce0-503c-4ddf-bb0c-82063c84417c" path="/var/lib/kubelet/pods/da7a7ce0-503c-4ddf-bb0c-82063c84417c/volumes" Sep 15 12:43:46.076113 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:46.076076 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm"] Sep 15 12:43:46.076606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:46.076351 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" containerID="cri-o://9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6" gracePeriod=30 Sep 15 12:43:51.028576 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.028540 2550 patch_prober.go:28] interesting pod/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 12:43:51.028916 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.028604 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 12:43:51.369842 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.369815 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:43:51.484252 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484214 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istiod-ca-cert\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484279 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-envoy\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484300 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-token\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484342 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-socket\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484366 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-data\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484420 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-credential-socket\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484457 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2qnv\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-kube-api-access-d2qnv\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484517 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-certs\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484546 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-podinfo\") pod \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\" (UID: \"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8\") " Sep 15 12:43:51.484799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484688 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:43:51.484799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484771 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:43:51.485000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484849 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:43:51.485000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484724 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:43:51.485000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484976 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:43:51.485000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.484986 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.485184 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.485007 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.485184 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.485021 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.487112 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.487081 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:43:51.487202 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.487154 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:43:51.487202 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.487168 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 12:43:51.487202 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.487180 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-kube-api-access-d2qnv" (OuterVolumeSpecName: "kube-api-access-d2qnv") pod "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" (UID: "fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8"). InnerVolumeSpecName "kube-api-access-d2qnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:43:51.586417 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.586366 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.586417 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.586410 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.586417 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.586422 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.586649 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.586434 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.586649 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.586445 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.586649 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.586455 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2qnv\" (UniqueName: \"kubernetes.io/projected/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8-kube-api-access-d2qnv\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:43:51.876985 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.876947 2550 generic.go:356] "Generic (PLEG): container finished" podID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerID="9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6" exitCode=0 Sep 15 12:43:51.877186 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.876999 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" Sep 15 12:43:51.877186 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.877004 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" event={"ID":"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8","Type":"ContainerDied","Data":"9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6"} Sep 15 12:43:51.877186 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.877087 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm" event={"ID":"fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8","Type":"ContainerDied","Data":"20c98defd8cdd34aba394d9c499ca3532d1ddd94594023689ad9230ecb9dc2c8"} Sep 15 12:43:51.877186 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.877119 2550 scope.go:117] "RemoveContainer" containerID="9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6" Sep 15 12:43:51.886844 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.886828 2550 scope.go:117] "RemoveContainer" containerID="c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759" Sep 15 12:43:51.894774 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.894758 2550 scope.go:117] "RemoveContainer" containerID="9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6" Sep 15 12:43:51.895036 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:43:51.895019 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6\": container with ID starting with 9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6 not found: ID does not exist" containerID="9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6" Sep 15 12:43:51.895092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.895045 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6"} err="failed to get container status \"9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6\": rpc error: code = NotFound desc = could not find container \"9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6\": container with ID starting with 9efa3859b07c778b772e6c6e03a671af2c48a1ca285731a2663186fb34967df6 not found: ID does not exist" Sep 15 12:43:51.895092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.895061 2550 scope.go:117] "RemoveContainer" containerID="c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759" Sep 15 12:43:51.895304 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:43:51.895287 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759\": container with ID starting with c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759 not found: ID does not exist" containerID="c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759" Sep 15 12:43:51.895357 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.895313 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759"} err="failed to get container status \"c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759\": rpc error: code = NotFound desc = could not find container \"c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759\": container with ID starting with c92345be2d34f5ac6a5fb6daba14d962fc4301cbde88f212e8bf4b13f1454759 not found: ID does not exist" Sep 15 12:43:51.898498 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.898476 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm"] Sep 15 12:43:51.902616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:51.902596 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b7e2d651-openshift-default-664bbf6468-66rgm"] Sep 15 12:43:53.154477 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:43:53.154445 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" path="/var/lib/kubelet/pods/fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8/volumes" Sep 15 12:44:00.408485 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.408425 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh"] Sep 15 12:44:00.409151 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409128 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" containerName="payload-pre-processing" Sep 15 12:44:00.409229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409153 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" containerName="payload-pre-processing" Sep 15 12:44:00.409229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409199 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" Sep 15 12:44:00.409229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409209 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" Sep 15 12:44:00.409229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409220 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3162d216-c268-4929-b6aa-92f00198c2f5" containerName="revoke-keys" Sep 15 12:44:00.409399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409230 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3162d216-c268-4929-b6aa-92f00198c2f5" containerName="revoke-keys" Sep 15 12:44:00.409399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409241 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7a7ce0-503c-4ddf-bb0c-82063c84417c" containerName="payload-processing" Sep 15 12:44:00.409399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409251 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7a7ce0-503c-4ddf-bb0c-82063c84417c" containerName="payload-processing" Sep 15 12:44:00.409399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409272 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" Sep 15 12:44:00.409399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409281 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" Sep 15 12:44:00.409399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409292 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0481b781-9ee6-4890-b6ab-067e6a8e2ba3" containerName="revoke-keys" Sep 15 12:44:00.409399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409300 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0481b781-9ee6-4890-b6ab-067e6a8e2ba3" containerName="revoke-keys" Sep 15 12:44:00.410012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409411 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d30b9d2-8953-450e-b8ea-ff57f2a7e49f" containerName="payload-pre-processing" Sep 15 12:44:00.410012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409422 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" Sep 15 12:44:00.410012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409432 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3162d216-c268-4929-b6aa-92f00198c2f5" containerName="revoke-keys" Sep 15 12:44:00.410012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409444 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0481b781-9ee6-4890-b6ab-067e6a8e2ba3" containerName="revoke-keys" Sep 15 12:44:00.410012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409453 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0ceab7-b8ed-4da0-b2a5-3c65efb146c8" containerName="istio-proxy" Sep 15 12:44:00.410012 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.409464 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7a7ce0-503c-4ddf-bb0c-82063c84417c" containerName="payload-processing" Sep 15 12:44:00.414422 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.414403 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.416779 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.416758 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a1ba7b-openshift-default-dockercfg-8fvbf\"" Sep 15 12:44:00.424722 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.424699 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh"] Sep 15 12:44:00.472537 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472488 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-data\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472551 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-envoy\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472575 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-token\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472610 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-socket\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472689 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-certs\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472734 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5669a06-72ff-4194-9610-a5852ba6c6f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472755 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-credential-socket\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472787 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr222\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-kube-api-access-lr222\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.472948 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.472860 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-podinfo\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574149 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574104 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-socket\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574476 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574450 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-certs\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574661 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574633 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-socket\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574786 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574637 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5669a06-72ff-4194-9610-a5852ba6c6f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574786 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574713 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-credential-socket\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574786 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574767 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr222\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-kube-api-access-lr222\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574828 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-podinfo\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574825 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-certs\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.574973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574948 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-data\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.575132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.574999 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-envoy\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.575132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.575046 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-token\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.575489 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.575462 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-data\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.575580 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.575486 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5669a06-72ff-4194-9610-a5852ba6c6f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.575580 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.575556 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-credential-socket\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.577373 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.577346 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-envoy\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.577564 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.577543 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-podinfo\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.585111 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.585086 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-token\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.585473 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.585455 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr222\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-kube-api-access-lr222\") pod \"e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:00.726157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:00.726062 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:01.002401 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:01.002349 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh"] Sep 15 12:44:01.004329 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:44:01.004304 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5669a06_72ff_4194_9610_a5852ba6c6f8.slice/crio-eb090540250039ec29637315a1d31f1c57d2b0c8a8db699ade64b8f40e7f931d WatchSource:0}: Error finding container eb090540250039ec29637315a1d31f1c57d2b0c8a8db699ade64b8f40e7f931d: Status 404 returned error can't find the container with id eb090540250039ec29637315a1d31f1c57d2b0c8a8db699ade64b8f40e7f931d Sep 15 12:44:01.006680 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:01.006634 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:01.006751 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:01.006735 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:01.006799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:01.006783 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:01.939411 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:01.939363 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" event={"ID":"e5669a06-72ff-4194-9610-a5852ba6c6f8","Type":"ContainerStarted","Data":"0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14"} Sep 15 12:44:01.939820 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:01.939436 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" event={"ID":"e5669a06-72ff-4194-9610-a5852ba6c6f8","Type":"ContainerStarted","Data":"eb090540250039ec29637315a1d31f1c57d2b0c8a8db699ade64b8f40e7f931d"} Sep 15 12:44:01.961262 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:01.961216 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podStartSLOduration=1.961201014 podStartE2EDuration="1.961201014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:44:01.95858931 +0000 UTC m=+1123.313435645" watchObservedRunningTime="2026-09-15 12:44:01.961201014 +0000 UTC m=+1123.316047336" Sep 15 12:44:02.726900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:02.726867 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:02.732183 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:02.732155 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:02.944871 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:02.944841 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:02.945886 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:02.945864 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:18.896562 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.896510 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v"] Sep 15 12:44:18.900468 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.900447 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.903085 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.903063 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-37b6a5-openshift-default-dockercfg-9dggs\"" Sep 15 12:44:18.913225 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.913203 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v"] Sep 15 12:44:18.964827 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.964783 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba2874d-e716-4c90-aea6-59791f2af19c-istio-podinfo\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.964834 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-credential-socket\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.964924 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-socket\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.964984 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba2874d-e716-4c90-aea6-59791f2af19c-istiod-ca-cert\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.965014 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-istio-token\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.965058 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-data\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.965094 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-certs\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965132 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.965117 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-envoy\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:18.965289 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:18.965163 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7krx\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-kube-api-access-v7krx\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066062 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066013 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba2874d-e716-4c90-aea6-59791f2af19c-istiod-ca-cert\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066062 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066065 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-istio-token\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066110 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-data\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066149 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-certs\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066174 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-envoy\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066206 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7krx\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-kube-api-access-v7krx\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066302 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba2874d-e716-4c90-aea6-59791f2af19c-istio-podinfo\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066332 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-credential-socket\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066678 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066400 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-socket\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066678 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066662 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba2874d-e716-4c90-aea6-59791f2af19c-istiod-ca-cert\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.066795 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.066782 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-socket\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.067080 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.067042 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-data\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.067315 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.067298 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-certs\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.067460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.067436 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-credential-socket\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.071025 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.068904 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-envoy\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.071025 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.069420 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba2874d-e716-4c90-aea6-59791f2af19c-istio-podinfo\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.074338 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.074304 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-istio-token\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.074451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.074344 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7krx\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-kube-api-access-v7krx\") pod \"e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.215901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.215823 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-37b6a5-openshift-default-dockercfg-9dggs\"" Sep 15 12:44:19.223972 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.223952 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:19.529005 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.528974 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v"] Sep 15 12:44:19.530615 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:44:19.530589 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba2874d_e716_4c90_aea6_59791f2af19c.slice/crio-6040462713b1ef62bc1c30d8888fb27adc7d47c86d4b43048fac1bc7f6d5cfc2 WatchSource:0}: Error finding container 6040462713b1ef62bc1c30d8888fb27adc7d47c86d4b43048fac1bc7f6d5cfc2: Status 404 returned error can't find the container with id 6040462713b1ef62bc1c30d8888fb27adc7d47c86d4b43048fac1bc7f6d5cfc2 Sep 15 12:44:19.533052 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.533020 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:19.533115 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.533100 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:19.533163 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:19.533149 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:20.066330 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:20.066213 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" event={"ID":"dba2874d-e716-4c90-aea6-59791f2af19c","Type":"ContainerStarted","Data":"3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753"} Sep 15 12:44:20.066330 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:20.066287 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" event={"ID":"dba2874d-e716-4c90-aea6-59791f2af19c","Type":"ContainerStarted","Data":"6040462713b1ef62bc1c30d8888fb27adc7d47c86d4b43048fac1bc7f6d5cfc2"} Sep 15 12:44:20.086066 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:20.086020 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" podStartSLOduration=2.086003176 podStartE2EDuration="2.086003176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:44:20.084245574 +0000 UTC m=+1141.439091898" watchObservedRunningTime="2026-09-15 12:44:20.086003176 +0000 UTC m=+1141.440849498" Sep 15 12:44:20.224804 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:20.224754 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:20.229688 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:20.229666 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:21.074721 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:21.074678 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:21.080804 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:21.080777 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:44:27.120070 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:27.120035 2550 generic.go:356] "Generic (PLEG): container finished" podID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerID="0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14" exitCode=0 Sep 15 12:44:27.120617 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:27.120113 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" event={"ID":"e5669a06-72ff-4194-9610-a5852ba6c6f8","Type":"ContainerDied","Data":"0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14"} Sep 15 12:44:27.120703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:27.120683 2550 scope.go:117] "RemoveContainer" containerID="0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14" Sep 15 12:44:27.121872 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:27.121840 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:27.121981 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:27.121921 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:27.121981 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:27.121967 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:44:28.127858 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:28.127815 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" event={"ID":"e5669a06-72ff-4194-9610-a5852ba6c6f8","Type":"ContainerStarted","Data":"e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651"} Sep 15 12:44:28.726357 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:28.726318 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:28.727313 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:28.727288 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 15 12:44:28.727410 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:28.727339 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 15 12:44:29.726936 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:29.726897 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 15 12:44:29.727327 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:29.726954 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 15 12:44:30.727432 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:30.727360 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 15 12:44:30.727822 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:30.727476 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 15 12:44:31.726570 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:31.726534 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 15 12:44:31.726739 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:31.726586 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 15 12:44:32.726829 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:32.726790 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 15 12:44:32.727268 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:32.726845 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 15 12:44:32.945259 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:32.945217 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:33.731326 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:33.731297 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:33.732246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:33.732229 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:44:42.307188 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.307097 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6"] Sep 15 12:44:42.312019 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.311995 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.314592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.314566 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d9e7a1\"" Sep 15 12:44:42.314716 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.314592 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d9e7a1-dockercfg-r85cp\"" Sep 15 12:44:42.320860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.320836 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6"] Sep 15 12:44:42.342650 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.342618 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l"] Sep 15 12:44:42.346329 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.346306 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.357757 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.357737 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l"] Sep 15 12:44:42.399503 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.399466 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7btkt\" (UniqueName: \"kubernetes.io/projected/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-kube-api-access-7btkt\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.399886 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.399859 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.500564 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.500533 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.500741 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.500614 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqznh\" (UniqueName: \"kubernetes.io/projected/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-kube-api-access-gqznh\") pod \"payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.500741 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.500637 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.500741 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.500697 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7btkt\" (UniqueName: \"kubernetes.io/projected/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-kube-api-access-7btkt\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.501223 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.501201 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.510677 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.510653 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7btkt\" (UniqueName: \"kubernetes.io/projected/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-kube-api-access-7btkt\") pod \"payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.601664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.601573 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqznh\" (UniqueName: \"kubernetes.io/projected/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-kube-api-access-gqznh\") pod \"payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.601664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.601625 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.602344 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.602323 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.609226 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.609203 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqznh\" (UniqueName: \"kubernetes.io/projected/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-kube-api-access-gqznh\") pod \"payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.624083 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.624055 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:42.658529 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.658498 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:42.902520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.902493 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6"] Sep 15 12:44:42.904347 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:44:42.904321 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f068d9_8901_4b0a_aff1_ae6e45553cc3.slice/crio-93ea3a90014eb2497f97e0dda9cd985131bf742bde32a82bbb769ec1b244a3d0 WatchSource:0}: Error finding container 93ea3a90014eb2497f97e0dda9cd985131bf742bde32a82bbb769ec1b244a3d0: Status 404 returned error can't find the container with id 93ea3a90014eb2497f97e0dda9cd985131bf742bde32a82bbb769ec1b244a3d0 Sep 15 12:44:42.948060 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:42.948035 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l"] Sep 15 12:44:42.950511 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:44:42.950484 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb367ea_98ac_4d2e_9d20_fdd0b0e9148a.slice/crio-a46bc022dcb83f7277ebd41d5db0d2c4019d873d44e982554a7c619814555cf2 WatchSource:0}: Error finding container a46bc022dcb83f7277ebd41d5db0d2c4019d873d44e982554a7c619814555cf2: Status 404 returned error can't find the container with id a46bc022dcb83f7277ebd41d5db0d2c4019d873d44e982554a7c619814555cf2 Sep 15 12:44:43.228065 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.227962 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" event={"ID":"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a","Type":"ContainerStarted","Data":"64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6"} Sep 15 12:44:43.228238 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.228109 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:43.228238 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.228165 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" event={"ID":"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a","Type":"ContainerStarted","Data":"a46bc022dcb83f7277ebd41d5db0d2c4019d873d44e982554a7c619814555cf2"} Sep 15 12:44:43.229746 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.229723 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" event={"ID":"d3f068d9-8901-4b0a-aff1-ae6e45553cc3","Type":"ContainerStarted","Data":"08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5"} Sep 15 12:44:43.229855 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.229764 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" event={"ID":"d3f068d9-8901-4b0a-aff1-ae6e45553cc3","Type":"ContainerStarted","Data":"93ea3a90014eb2497f97e0dda9cd985131bf742bde32a82bbb769ec1b244a3d0"} Sep 15 12:44:43.229855 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.229804 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:44:43.247241 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.247190 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" podStartSLOduration=1.247176722 podStartE2EDuration="1.247176722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:44:43.245619385 +0000 UTC m=+1164.600465710" watchObservedRunningTime="2026-09-15 12:44:43.247176722 +0000 UTC m=+1164.602023044" Sep 15 12:44:43.263288 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:43.263234 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" podStartSLOduration=1.263220102 podStartE2EDuration="1.263220102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:44:43.261081437 +0000 UTC m=+1164.615927997" watchObservedRunningTime="2026-09-15 12:44:43.263220102 +0000 UTC m=+1164.618066426" Sep 15 12:44:54.237914 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:54.237881 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:44:54.238420 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:44:54.238089 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:45:00.173484 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.173444 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m"] Sep 15 12:45:00.178042 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.178021 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb"] Sep 15 12:45:00.178149 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.178127 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" Sep 15 12:45:00.181375 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.181150 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:45:00.181667 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.181649 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd"] Sep 15 12:45:00.181760 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.181743 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" Sep 15 12:45:00.185172 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.185147 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" Sep 15 12:45:00.188621 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.188594 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb"] Sep 15 12:45:00.192894 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.192855 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m"] Sep 15 12:45:00.194055 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.193897 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" Sep 15 12:45:00.195263 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.194894 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd"] Sep 15 12:45:00.202708 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.202687 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7"] Sep 15 12:45:00.206838 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.206818 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" Sep 15 12:45:00.206956 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.206926 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" Sep 15 12:45:00.217032 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.217004 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7"] Sep 15 12:45:00.249657 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.249630 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" Sep 15 12:45:00.259289 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.259266 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" Sep 15 12:45:00.492971 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.492944 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m"] Sep 15 12:45:00.495631 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:45:00.495600 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3675cf_8f8a_4a23_a956_6fd1ea98211f.slice/crio-9b586d273f695dba59dc34e29cbc81e65e85ec657357b69391abdee377930e59 WatchSource:0}: Error finding container 9b586d273f695dba59dc34e29cbc81e65e85ec657357b69391abdee377930e59: Status 404 returned error can't find the container with id 9b586d273f695dba59dc34e29cbc81e65e85ec657357b69391abdee377930e59 Sep 15 12:45:00.516667 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.516644 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb"] Sep 15 12:45:00.518716 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:45:00.518679 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24db67b2_af79_4efd_8ec5_11cf996665a9.slice/crio-3b0d01c5c3ff80634d83ab4d72e05d1ab05b7695c097a2748b5dd774b1ae910f WatchSource:0}: Error finding container 3b0d01c5c3ff80634d83ab4d72e05d1ab05b7695c097a2748b5dd774b1ae910f: Status 404 returned error can't find the container with id 3b0d01c5c3ff80634d83ab4d72e05d1ab05b7695c097a2748b5dd774b1ae910f Sep 15 12:45:00.553599 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.553571 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd"] Sep 15 12:45:00.555175 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:45:00.555151 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c144fe_6005_41e5_9faa_d77057e91ecc.slice/crio-5fb332a421850bbd740b19d50eb5e641053b87a9d5d201cf7eabcf8f913ba77b WatchSource:0}: Error finding container 5fb332a421850bbd740b19d50eb5e641053b87a9d5d201cf7eabcf8f913ba77b: Status 404 returned error can't find the container with id 5fb332a421850bbd740b19d50eb5e641053b87a9d5d201cf7eabcf8f913ba77b Sep 15 12:45:00.569125 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:00.569100 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7"] Sep 15 12:45:00.570711 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:45:00.570690 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3945a01_d784_4175_97a5_de579badbba3.slice/crio-cd7a4c026170b96184748ddd4e9845383173cec9787ed8aa338b39defd757021 WatchSource:0}: Error finding container cd7a4c026170b96184748ddd4e9845383173cec9787ed8aa338b39defd757021: Status 404 returned error can't find the container with id cd7a4c026170b96184748ddd4e9845383173cec9787ed8aa338b39defd757021 Sep 15 12:45:01.345113 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.345078 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" event={"ID":"24db67b2-af79-4efd-8ec5-11cf996665a9","Type":"ContainerStarted","Data":"0873735986d0dd54d40e5b612435ca83e13937e0097e07feb7f250b372ba52a3"} Sep 15 12:45:01.345570 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.345142 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" event={"ID":"24db67b2-af79-4efd-8ec5-11cf996665a9","Type":"ContainerStarted","Data":"3b0d01c5c3ff80634d83ab4d72e05d1ab05b7695c097a2748b5dd774b1ae910f"} Sep 15 12:45:01.346650 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.346623 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" event={"ID":"d3945a01-d784-4175-97a5-de579badbba3","Type":"ContainerStarted","Data":"e527454ff2662bf4b9cfa7c2a5190f5434ba4ce9456a371de6ae4e3ea0b4013a"} Sep 15 12:45:01.346778 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.346668 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" event={"ID":"d3945a01-d784-4175-97a5-de579badbba3","Type":"ContainerStarted","Data":"cd7a4c026170b96184748ddd4e9845383173cec9787ed8aa338b39defd757021"} Sep 15 12:45:01.348091 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.348065 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" event={"ID":"37c144fe-6005-41e5-9faa-d77057e91ecc","Type":"ContainerStarted","Data":"4c0a1d8df027382138a28f361d09b23b89e9be835616382bc2a72a98c651b799"} Sep 15 12:45:01.348188 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.348113 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" event={"ID":"37c144fe-6005-41e5-9faa-d77057e91ecc","Type":"ContainerStarted","Data":"5fb332a421850bbd740b19d50eb5e641053b87a9d5d201cf7eabcf8f913ba77b"} Sep 15 12:45:01.349665 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.349644 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" event={"ID":"6c3675cf-8f8a-4a23-a956-6fd1ea98211f","Type":"ContainerStarted","Data":"40568d446b6dcaba15fcdc95ab6e73c8c30766f91f130828bd3c0b6fe66801ec"} Sep 15 12:45:01.349766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.349680 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" event={"ID":"6c3675cf-8f8a-4a23-a956-6fd1ea98211f","Type":"ContainerStarted","Data":"9b586d273f695dba59dc34e29cbc81e65e85ec657357b69391abdee377930e59"} Sep 15 12:45:01.360508 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.360466 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" podStartSLOduration=1.360452198 podStartE2EDuration="1.360452198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:01.358057788 +0000 UTC m=+1182.712904112" watchObservedRunningTime="2026-09-15 12:45:01.360452198 +0000 UTC m=+1182.715298521" Sep 15 12:45:01.371285 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.371243 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" podStartSLOduration=1.3712308420000001 podStartE2EDuration="1.371230842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:01.370294088 +0000 UTC m=+1182.725140412" watchObservedRunningTime="2026-09-15 12:45:01.371230842 +0000 UTC m=+1182.726077164" Sep 15 12:45:01.382976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.382932 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" podStartSLOduration=1.382914013 podStartE2EDuration="1.382914013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:01.382721258 +0000 UTC m=+1182.737567582" watchObservedRunningTime="2026-09-15 12:45:01.382914013 +0000 UTC m=+1182.737760417" Sep 15 12:45:01.398142 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:01.398090 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" podStartSLOduration=1.398075354 podStartE2EDuration="1.398075354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:01.395492887 +0000 UTC m=+1182.750339209" watchObservedRunningTime="2026-09-15 12:45:01.398075354 +0000 UTC m=+1182.752921677" Sep 15 12:45:02.361542 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.361510 2550 generic.go:356] "Generic (PLEG): container finished" podID="6c3675cf-8f8a-4a23-a956-6fd1ea98211f" containerID="40568d446b6dcaba15fcdc95ab6e73c8c30766f91f130828bd3c0b6fe66801ec" exitCode=0 Sep 15 12:45:02.361971 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.361592 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" event={"ID":"6c3675cf-8f8a-4a23-a956-6fd1ea98211f","Type":"ContainerDied","Data":"40568d446b6dcaba15fcdc95ab6e73c8c30766f91f130828bd3c0b6fe66801ec"} Sep 15 12:45:02.363278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.363253 2550 generic.go:356] "Generic (PLEG): container finished" podID="24db67b2-af79-4efd-8ec5-11cf996665a9" containerID="0873735986d0dd54d40e5b612435ca83e13937e0097e07feb7f250b372ba52a3" exitCode=0 Sep 15 12:45:02.363418 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.363307 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" event={"ID":"24db67b2-af79-4efd-8ec5-11cf996665a9","Type":"ContainerDied","Data":"0873735986d0dd54d40e5b612435ca83e13937e0097e07feb7f250b372ba52a3"} Sep 15 12:45:02.364790 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.364767 2550 generic.go:356] "Generic (PLEG): container finished" podID="d3945a01-d784-4175-97a5-de579badbba3" containerID="e527454ff2662bf4b9cfa7c2a5190f5434ba4ce9456a371de6ae4e3ea0b4013a" exitCode=0 Sep 15 12:45:02.364888 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.364847 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" event={"ID":"d3945a01-d784-4175-97a5-de579badbba3","Type":"ContainerDied","Data":"e527454ff2662bf4b9cfa7c2a5190f5434ba4ce9456a371de6ae4e3ea0b4013a"} Sep 15 12:45:02.366407 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.366371 2550 generic.go:356] "Generic (PLEG): container finished" podID="37c144fe-6005-41e5-9faa-d77057e91ecc" containerID="4c0a1d8df027382138a28f361d09b23b89e9be835616382bc2a72a98c651b799" exitCode=0 Sep 15 12:45:02.366505 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:02.366403 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" event={"ID":"37c144fe-6005-41e5-9faa-d77057e91ecc","Type":"ContainerDied","Data":"4c0a1d8df027382138a28f361d09b23b89e9be835616382bc2a72a98c651b799"} Sep 15 12:45:03.538303 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:03.538277 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" Sep 15 12:45:03.588472 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:03.588435 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" Sep 15 12:45:03.625286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:03.625259 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" Sep 15 12:45:03.628512 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:03.628492 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" Sep 15 12:45:04.386945 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.386917 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" Sep 15 12:45:04.387170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.386954 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7" event={"ID":"d3945a01-d784-4175-97a5-de579badbba3","Type":"ContainerDied","Data":"cd7a4c026170b96184748ddd4e9845383173cec9787ed8aa338b39defd757021"} Sep 15 12:45:04.387170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.386999 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7a4c026170b96184748ddd4e9845383173cec9787ed8aa338b39defd757021" Sep 15 12:45:04.388358 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.388339 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" Sep 15 12:45:04.388460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.388416 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd" event={"ID":"37c144fe-6005-41e5-9faa-d77057e91ecc","Type":"ContainerDied","Data":"5fb332a421850bbd740b19d50eb5e641053b87a9d5d201cf7eabcf8f913ba77b"} Sep 15 12:45:04.388460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.388453 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fb332a421850bbd740b19d50eb5e641053b87a9d5d201cf7eabcf8f913ba77b" Sep 15 12:45:04.390027 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.390002 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" event={"ID":"6c3675cf-8f8a-4a23-a956-6fd1ea98211f","Type":"ContainerDied","Data":"9b586d273f695dba59dc34e29cbc81e65e85ec657357b69391abdee377930e59"} Sep 15 12:45:04.390191 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.390175 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b586d273f695dba59dc34e29cbc81e65e85ec657357b69391abdee377930e59" Sep 15 12:45:04.390446 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.390197 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m" Sep 15 12:45:04.391778 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.391753 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" event={"ID":"24db67b2-af79-4efd-8ec5-11cf996665a9","Type":"ContainerDied","Data":"3b0d01c5c3ff80634d83ab4d72e05d1ab05b7695c097a2748b5dd774b1ae910f"} Sep 15 12:45:04.391862 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.391774 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb" Sep 15 12:45:04.391862 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:04.391790 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b0d01c5c3ff80634d83ab4d72e05d1ab05b7695c097a2748b5dd774b1ae910f" Sep 15 12:45:19.172792 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:19.172753 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:45:19.177635 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:19.177613 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:45:19.262194 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:19.262163 2550 scope.go:117] "RemoveContainer" containerID="aa87f90511a3047c73c7f702436d95f784f6d1f1a36a9f16b22942c6663086b5" Sep 15 12:45:26.259331 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.259280 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks"] Sep 15 12:45:26.260106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260082 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c144fe-6005-41e5-9faa-d77057e91ecc" containerName="cleanup" Sep 15 12:45:26.260207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260110 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c144fe-6005-41e5-9faa-d77057e91ecc" containerName="cleanup" Sep 15 12:45:26.260207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260125 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3675cf-8f8a-4a23-a956-6fd1ea98211f" containerName="cleanup" Sep 15 12:45:26.260207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260134 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3675cf-8f8a-4a23-a956-6fd1ea98211f" containerName="cleanup" Sep 15 12:45:26.260207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260144 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3945a01-d784-4175-97a5-de579badbba3" containerName="cleanup" Sep 15 12:45:26.260207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260154 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3945a01-d784-4175-97a5-de579badbba3" containerName="cleanup" Sep 15 12:45:26.260207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260173 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24db67b2-af79-4efd-8ec5-11cf996665a9" containerName="cleanup" Sep 15 12:45:26.260207 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260183 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="24db67b2-af79-4efd-8ec5-11cf996665a9" containerName="cleanup" Sep 15 12:45:26.260504 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260302 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3945a01-d784-4175-97a5-de579badbba3" containerName="cleanup" Sep 15 12:45:26.260504 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260315 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3675cf-8f8a-4a23-a956-6fd1ea98211f" containerName="cleanup" Sep 15 12:45:26.260504 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260329 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c144fe-6005-41e5-9faa-d77057e91ecc" containerName="cleanup" Sep 15 12:45:26.260504 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.260337 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="24db67b2-af79-4efd-8ec5-11cf996665a9" containerName="cleanup" Sep 15 12:45:26.265447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.265423 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.267749 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.267726 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-88615e8e\"" Sep 15 12:45:26.267900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.267880 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-88615e8e-dockercfg-8sctx\"" Sep 15 12:45:26.268594 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.268558 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2st7\" (UniqueName: \"kubernetes.io/projected/a2669a2b-28ba-4856-a745-9785f1edff1b-kube-api-access-p2st7\") pod \"payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.268706 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.268637 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2669a2b-28ba-4856-a745-9785f1edff1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.270014 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.269991 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks"] Sep 15 12:45:26.365317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.365280 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9"] Sep 15 12:45:26.369030 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.369012 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.369243 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.369216 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2669a2b-28ba-4856-a745-9785f1edff1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.369342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.369329 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2st7\" (UniqueName: \"kubernetes.io/projected/a2669a2b-28ba-4856-a745-9785f1edff1b-kube-api-access-p2st7\") pod \"payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.370008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.369987 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2669a2b-28ba-4856-a745-9785f1edff1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.379133 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.379104 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9"] Sep 15 12:45:26.380411 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.380369 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2st7\" (UniqueName: \"kubernetes.io/projected/a2669a2b-28ba-4856-a745-9785f1edff1b-kube-api-access-p2st7\") pod \"payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.470275 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.470224 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf657\" (UniqueName: \"kubernetes.io/projected/e65aa87c-948a-4d79-86d5-feb5b47b41d0-kube-api-access-tf657\") pod \"payload-processing-e2e-mt-88615e8e-5694874589-blkp9\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.470275 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.470284 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65aa87c-948a-4d79-86d5-feb5b47b41d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-88615e8e-5694874589-blkp9\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.571143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.571042 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf657\" (UniqueName: \"kubernetes.io/projected/e65aa87c-948a-4d79-86d5-feb5b47b41d0-kube-api-access-tf657\") pod \"payload-processing-e2e-mt-88615e8e-5694874589-blkp9\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.571143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.571103 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65aa87c-948a-4d79-86d5-feb5b47b41d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-88615e8e-5694874589-blkp9\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.571810 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.571787 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65aa87c-948a-4d79-86d5-feb5b47b41d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-88615e8e-5694874589-blkp9\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.578415 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.578395 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:26.578767 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.578746 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf657\" (UniqueName: \"kubernetes.io/projected/e65aa87c-948a-4d79-86d5-feb5b47b41d0-kube-api-access-tf657\") pod \"payload-processing-e2e-mt-88615e8e-5694874589-blkp9\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.680908 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.680866 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:26.852169 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.852138 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks"] Sep 15 12:45:26.853106 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:45:26.853080 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2669a2b_28ba_4856_a745_9785f1edff1b.slice/crio-7ce744c0b12526fac748434a817382cece82904512935bed17b489d2da410f5f WatchSource:0}: Error finding container 7ce744c0b12526fac748434a817382cece82904512935bed17b489d2da410f5f: Status 404 returned error can't find the container with id 7ce744c0b12526fac748434a817382cece82904512935bed17b489d2da410f5f Sep 15 12:45:26.951350 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:26.951321 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9"] Sep 15 12:45:26.953021 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:45:26.952996 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65aa87c_948a_4d79_86d5_feb5b47b41d0.slice/crio-898c61c0cbe5bae378c546a3899066f5d9181a756e928122a70b4f72f8f73178 WatchSource:0}: Error finding container 898c61c0cbe5bae378c546a3899066f5d9181a756e928122a70b4f72f8f73178: Status 404 returned error can't find the container with id 898c61c0cbe5bae378c546a3899066f5d9181a756e928122a70b4f72f8f73178 Sep 15 12:45:27.536227 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.536189 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" event={"ID":"e65aa87c-948a-4d79-86d5-feb5b47b41d0","Type":"ContainerStarted","Data":"ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da"} Sep 15 12:45:27.536745 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.536248 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" event={"ID":"e65aa87c-948a-4d79-86d5-feb5b47b41d0","Type":"ContainerStarted","Data":"898c61c0cbe5bae378c546a3899066f5d9181a756e928122a70b4f72f8f73178"} Sep 15 12:45:27.536745 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.536402 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:27.538049 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.538026 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" event={"ID":"a2669a2b-28ba-4856-a745-9785f1edff1b","Type":"ContainerStarted","Data":"85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb"} Sep 15 12:45:27.538177 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.538072 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" event={"ID":"a2669a2b-28ba-4856-a745-9785f1edff1b","Type":"ContainerStarted","Data":"7ce744c0b12526fac748434a817382cece82904512935bed17b489d2da410f5f"} Sep 15 12:45:27.538177 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.538098 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:27.554831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.554782 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" podStartSLOduration=1.554766517 podStartE2EDuration="1.554766517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:27.552412914 +0000 UTC m=+1208.907259238" watchObservedRunningTime="2026-09-15 12:45:27.554766517 +0000 UTC m=+1208.909612883" Sep 15 12:45:27.567744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:27.567690 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" podStartSLOduration=1.5676719449999998 podStartE2EDuration="1.567671945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:45:27.566093361 +0000 UTC m=+1208.920939686" watchObservedRunningTime="2026-09-15 12:45:27.567671945 +0000 UTC m=+1208.922518267" Sep 15 12:45:38.545946 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:38.545912 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:45:38.546311 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:38.545981 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:45:43.327498 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:43.327458 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v"] Sep 15 12:45:43.327939 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:43.327710 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" podUID="dba2874d-e716-4c90-aea6-59791f2af19c" containerName="istio-proxy" containerID="cri-o://3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753" gracePeriod=30 Sep 15 12:45:48.595436 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.595359 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:45:48.663949 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.663909 2550 generic.go:356] "Generic (PLEG): container finished" podID="dba2874d-e716-4c90-aea6-59791f2af19c" containerID="3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753" exitCode=0 Sep 15 12:45:48.664157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.663960 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" Sep 15 12:45:48.664157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.663959 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" event={"ID":"dba2874d-e716-4c90-aea6-59791f2af19c","Type":"ContainerDied","Data":"3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753"} Sep 15 12:45:48.664157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.664023 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v" event={"ID":"dba2874d-e716-4c90-aea6-59791f2af19c","Type":"ContainerDied","Data":"6040462713b1ef62bc1c30d8888fb27adc7d47c86d4b43048fac1bc7f6d5cfc2"} Sep 15 12:45:48.664157 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.664073 2550 scope.go:117] "RemoveContainer" containerID="3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753" Sep 15 12:45:48.674290 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.674261 2550 scope.go:117] "RemoveContainer" containerID="3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753" Sep 15 12:45:48.674744 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:45:48.674719 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753\": container with ID starting with 3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753 not found: ID does not exist" containerID="3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753" Sep 15 12:45:48.674834 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.674753 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753"} err="failed to get container status \"3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753\": rpc error: code = NotFound desc = could not find container \"3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753\": container with ID starting with 3e69d1391ac638915d805df493965b8fcd610597e038f187d8cedd940cb77753 not found: ID does not exist" Sep 15 12:45:48.691098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691070 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-envoy\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691112 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba2874d-e716-4c90-aea6-59791f2af19c-istio-podinfo\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691130 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-istio-token\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691150 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7krx\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-kube-api-access-v7krx\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691176 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-data\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691211 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-credential-socket\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691511 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691265 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba2874d-e716-4c90-aea6-59791f2af19c-istiod-ca-cert\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691511 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691350 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-socket\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691511 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691406 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-certs\") pod \"dba2874d-e716-4c90-aea6-59791f2af19c\" (UID: \"dba2874d-e716-4c90-aea6-59791f2af19c\") " Sep 15 12:45:48.691680 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691612 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:45:48.691745 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691701 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba2874d-e716-4c90-aea6-59791f2af19c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:45:48.691977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691846 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:45:48.691977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691875 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-data" (OuterVolumeSpecName: "istio-data") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:45:48.691977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.691944 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:45:48.692197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.692053 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.692197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.692075 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.692197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.692090 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dba2874d-e716-4c90-aea6-59791f2af19c-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.692197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.692105 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.692197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.692119 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.693447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.693419 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dba2874d-e716-4c90-aea6-59791f2af19c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 12:45:48.693533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.693448 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-istio-token" (OuterVolumeSpecName: "istio-token") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:45:48.693579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.693528 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:45:48.693641 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.693621 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-kube-api-access-v7krx" (OuterVolumeSpecName: "kube-api-access-v7krx") pod "dba2874d-e716-4c90-aea6-59791f2af19c" (UID: "dba2874d-e716-4c90-aea6-59791f2af19c"). InnerVolumeSpecName "kube-api-access-v7krx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:45:48.795398 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.795292 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dba2874d-e716-4c90-aea6-59791f2af19c-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.795398 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.795330 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dba2874d-e716-4c90-aea6-59791f2af19c-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.795398 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.795345 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.795398 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.795360 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7krx\" (UniqueName: \"kubernetes.io/projected/dba2874d-e716-4c90-aea6-59791f2af19c-kube-api-access-v7krx\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:45:48.988101 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.988072 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v"] Sep 15 12:45:48.992355 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:48.992330 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-37b6a5-openshift-default-5575c4f57b-t878v"] Sep 15 12:45:49.155080 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:45:49.155046 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba2874d-e716-4c90-aea6-59791f2af19c" path="/var/lib/kubelet/pods/dba2874d-e716-4c90-aea6-59791f2af19c/volumes" Sep 15 12:46:08.287794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.287756 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl"] Sep 15 12:46:08.288260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.288247 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba2874d-e716-4c90-aea6-59791f2af19c" containerName="istio-proxy" Sep 15 12:46:08.288306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.288261 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba2874d-e716-4c90-aea6-59791f2af19c" containerName="istio-proxy" Sep 15 12:46:08.288341 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.288329 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba2874d-e716-4c90-aea6-59791f2af19c" containerName="istio-proxy" Sep 15 12:46:08.293193 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.293176 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" Sep 15 12:46:08.295746 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.295719 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:46:08.296078 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.296056 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:46:08.298340 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.298316 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl"] Sep 15 12:46:08.382748 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.382711 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9feb9b-b337-4311-b182-87f660c9836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl\" (UID: \"eb9feb9b-b337-4311-b182-87f660c9836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" Sep 15 12:46:08.483464 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.483427 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9feb9b-b337-4311-b182-87f660c9836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl\" (UID: \"eb9feb9b-b337-4311-b182-87f660c9836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" Sep 15 12:46:08.484013 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.483995 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9feb9b-b337-4311-b182-87f660c9836d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl\" (UID: \"eb9feb9b-b337-4311-b182-87f660c9836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" Sep 15 12:46:08.606410 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.606290 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" Sep 15 12:46:08.879357 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:08.879324 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl"] Sep 15 12:46:08.880972 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:46:08.880946 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9feb9b_b337_4311_b182_87f660c9836d.slice/crio-6d6e15ca41d29c554951161fc64a56432fef3dbde1fb9d30ebdef5a2ce95800e WatchSource:0}: Error finding container 6d6e15ca41d29c554951161fc64a56432fef3dbde1fb9d30ebdef5a2ce95800e: Status 404 returned error can't find the container with id 6d6e15ca41d29c554951161fc64a56432fef3dbde1fb9d30ebdef5a2ce95800e Sep 15 12:46:09.798895 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:09.798801 2550 generic.go:356] "Generic (PLEG): container finished" podID="eb9feb9b-b337-4311-b182-87f660c9836d" containerID="a2c9fd1048a5523d667e53606cbea372a25be04cdd0853318ea2ca9da5711abc" exitCode=0 Sep 15 12:46:09.799243 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:09.798892 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" event={"ID":"eb9feb9b-b337-4311-b182-87f660c9836d","Type":"ContainerDied","Data":"a2c9fd1048a5523d667e53606cbea372a25be04cdd0853318ea2ca9da5711abc"} Sep 15 12:46:09.799243 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:09.798940 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" event={"ID":"eb9feb9b-b337-4311-b182-87f660c9836d","Type":"ContainerStarted","Data":"6d6e15ca41d29c554951161fc64a56432fef3dbde1fb9d30ebdef5a2ce95800e"} Sep 15 12:46:10.957927 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:10.957899 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" Sep 15 12:46:11.008680 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:11.008635 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9feb9b-b337-4311-b182-87f660c9836d-maas-api-service-ca\") pod \"eb9feb9b-b337-4311-b182-87f660c9836d\" (UID: \"eb9feb9b-b337-4311-b182-87f660c9836d\") " Sep 15 12:46:11.009037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:11.009011 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb9feb9b-b337-4311-b182-87f660c9836d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb9feb9b-b337-4311-b182-87f660c9836d" (UID: "eb9feb9b-b337-4311-b182-87f660c9836d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:46:11.109497 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:11.109413 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb9feb9b-b337-4311-b182-87f660c9836d-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:11.811724 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:11.811690 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" Sep 15 12:46:11.811884 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:11.811730 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl" event={"ID":"eb9feb9b-b337-4311-b182-87f660c9836d","Type":"ContainerDied","Data":"6d6e15ca41d29c554951161fc64a56432fef3dbde1fb9d30ebdef5a2ce95800e"} Sep 15 12:46:11.811884 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:11.811781 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d6e15ca41d29c554951161fc64a56432fef3dbde1fb9d30ebdef5a2ce95800e" Sep 15 12:46:18.808347 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:18.808312 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd"] Sep 15 12:46:18.811805 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:18.811779 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-4609ba-29824605-q7wfd"] Sep 15 12:46:18.837185 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:18.837153 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj"] Sep 15 12:46:18.837520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:18.837463 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" podUID="d4618891-5a99-45ef-96e6-021c7806cb2b" containerName="payload-processing" containerID="cri-o://42aba35bd12fd4782bbdbfaf620b9569e665a6f69c40578a11d318737667ccf3" gracePeriod=30 Sep 15 12:46:18.846875 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:18.846842 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz"] Sep 15 12:46:18.847150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:18.847099 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" podUID="cca36773-1899-401b-bdb7-b311309903ad" containerName="payload-pre-processing" containerID="cri-o://8aedc89daf3d3c87d7486f04730317a9069ce0cbabae45c6e55578e4a734c9b5" gracePeriod=30 Sep 15 12:46:19.002059 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.002022 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl"] Sep 15 12:46:19.004416 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.004365 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-4609ba-kc8nl"] Sep 15 12:46:19.156352 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.156328 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:46:19.156743 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.156721 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37c144fe-6005-41e5-9faa-d77057e91ecc" path="/var/lib/kubelet/pods/37c144fe-6005-41e5-9faa-d77057e91ecc/volumes" Sep 15 12:46:19.157106 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.157086 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9feb9b-b337-4311-b182-87f660c9836d" path="/var/lib/kubelet/pods/eb9feb9b-b337-4311-b182-87f660c9836d/volumes" Sep 15 12:46:19.174148 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.174124 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:46:19.284793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.284761 2550 scope.go:117] "RemoveContainer" containerID="42aba35bd12fd4782bbdbfaf620b9569e665a6f69c40578a11d318737667ccf3" Sep 15 12:46:19.293140 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.293115 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st6gc\" (UniqueName: \"kubernetes.io/projected/d4618891-5a99-45ef-96e6-021c7806cb2b-kube-api-access-st6gc\") pod \"d4618891-5a99-45ef-96e6-021c7806cb2b\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " Sep 15 12:46:19.293258 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.293165 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvjtm\" (UniqueName: \"kubernetes.io/projected/cca36773-1899-401b-bdb7-b311309903ad-kube-api-access-kvjtm\") pod \"cca36773-1899-401b-bdb7-b311309903ad\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " Sep 15 12:46:19.293331 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.293261 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca36773-1899-401b-bdb7-b311309903ad-plugins-config-volume\") pod \"cca36773-1899-401b-bdb7-b311309903ad\" (UID: \"cca36773-1899-401b-bdb7-b311309903ad\") " Sep 15 12:46:19.293331 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.293282 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4618891-5a99-45ef-96e6-021c7806cb2b-plugins-config-volume\") pod \"d4618891-5a99-45ef-96e6-021c7806cb2b\" (UID: \"d4618891-5a99-45ef-96e6-021c7806cb2b\") " Sep 15 12:46:19.293683 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.293661 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca36773-1899-401b-bdb7-b311309903ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cca36773-1899-401b-bdb7-b311309903ad" (UID: "cca36773-1899-401b-bdb7-b311309903ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:46:19.293802 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.293708 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4618891-5a99-45ef-96e6-021c7806cb2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4618891-5a99-45ef-96e6-021c7806cb2b" (UID: "d4618891-5a99-45ef-96e6-021c7806cb2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:46:19.294508 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.294487 2550 scope.go:117] "RemoveContainer" containerID="2d81a85b308841bd4e3b5a564008d5cf15f8da260299a5d60496234ac6b8a77c" Sep 15 12:46:19.295482 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.295372 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4618891-5a99-45ef-96e6-021c7806cb2b-kube-api-access-st6gc" (OuterVolumeSpecName: "kube-api-access-st6gc") pod "d4618891-5a99-45ef-96e6-021c7806cb2b" (UID: "d4618891-5a99-45ef-96e6-021c7806cb2b"). InnerVolumeSpecName "kube-api-access-st6gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:46:19.295482 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.295420 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca36773-1899-401b-bdb7-b311309903ad-kube-api-access-kvjtm" (OuterVolumeSpecName: "kube-api-access-kvjtm") pod "cca36773-1899-401b-bdb7-b311309903ad" (UID: "cca36773-1899-401b-bdb7-b311309903ad"). InnerVolumeSpecName "kube-api-access-kvjtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:46:19.302334 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.302317 2550 scope.go:117] "RemoveContainer" containerID="8aedc89daf3d3c87d7486f04730317a9069ce0cbabae45c6e55578e4a734c9b5" Sep 15 12:46:19.394648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.394578 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca36773-1899-401b-bdb7-b311309903ad-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:19.394648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.394605 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4618891-5a99-45ef-96e6-021c7806cb2b-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:19.394648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.394614 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6gc\" (UniqueName: \"kubernetes.io/projected/d4618891-5a99-45ef-96e6-021c7806cb2b-kube-api-access-st6gc\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:19.394648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.394623 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvjtm\" (UniqueName: \"kubernetes.io/projected/cca36773-1899-401b-bdb7-b311309903ad-kube-api-access-kvjtm\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:19.865327 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.865292 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" event={"ID":"d4618891-5a99-45ef-96e6-021c7806cb2b","Type":"ContainerDied","Data":"42aba35bd12fd4782bbdbfaf620b9569e665a6f69c40578a11d318737667ccf3"} Sep 15 12:46:19.865327 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.865321 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" Sep 15 12:46:19.865699 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.865357 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" event={"ID":"d4618891-5a99-45ef-96e6-021c7806cb2b","Type":"ContainerDied","Data":"ec8ffef0b2be8d02c551afe7be2d8e6c3dcf0b471a0381cc5f191374ae1e612c"} Sep 15 12:46:19.865699 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.865321 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj" Sep 15 12:46:19.865699 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.865403 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" event={"ID":"cca36773-1899-401b-bdb7-b311309903ad","Type":"ContainerDied","Data":"8aedc89daf3d3c87d7486f04730317a9069ce0cbabae45c6e55578e4a734c9b5"} Sep 15 12:46:19.865699 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.865427 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz" event={"ID":"cca36773-1899-401b-bdb7-b311309903ad","Type":"ContainerDied","Data":"d1e71923677ff063bd9d46d15f8c11ab9156316217ebc28a13b5ba98cb495700"} Sep 15 12:46:19.913530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.913499 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz"] Sep 15 12:46:19.915776 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.915753 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4609ba-54db8bdbf8-dtzgz"] Sep 15 12:46:19.925040 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.925016 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj"] Sep 15 12:46:19.928002 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:19.927983 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4609ba-57946bbc7-mx7nj"] Sep 15 12:46:21.154944 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:21.154909 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca36773-1899-401b-bdb7-b311309903ad" path="/var/lib/kubelet/pods/cca36773-1899-401b-bdb7-b311309903ad/volumes" Sep 15 12:46:21.162133 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:21.155856 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4618891-5a99-45ef-96e6-021c7806cb2b" path="/var/lib/kubelet/pods/d4618891-5a99-45ef-96e6-021c7806cb2b/volumes" Sep 15 12:46:25.927486 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.927452 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2"] Sep 15 12:46:25.927927 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.927914 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9feb9b-b337-4311-b182-87f660c9836d" containerName="revoke-keys" Sep 15 12:46:25.927976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.927928 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9feb9b-b337-4311-b182-87f660c9836d" containerName="revoke-keys" Sep 15 12:46:25.927976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.927939 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4618891-5a99-45ef-96e6-021c7806cb2b" containerName="payload-processing" Sep 15 12:46:25.927976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.927944 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4618891-5a99-45ef-96e6-021c7806cb2b" containerName="payload-processing" Sep 15 12:46:25.927976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.927971 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca36773-1899-401b-bdb7-b311309903ad" containerName="payload-pre-processing" Sep 15 12:46:25.927976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.927978 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca36773-1899-401b-bdb7-b311309903ad" containerName="payload-pre-processing" Sep 15 12:46:25.928129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.928048 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca36773-1899-401b-bdb7-b311309903ad" containerName="payload-pre-processing" Sep 15 12:46:25.928129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.928055 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9feb9b-b337-4311-b182-87f660c9836d" containerName="revoke-keys" Sep 15 12:46:25.928129 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.928062 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4618891-5a99-45ef-96e6-021c7806cb2b" containerName="payload-processing" Sep 15 12:46:25.932881 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.932866 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" Sep 15 12:46:25.939188 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.939164 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:46:25.939741 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.939720 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:46:25.942800 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:25.942773 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2"] Sep 15 12:46:26.061306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.061267 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb4ca5-d116-4634-a743-1b068b5682ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2\" (UID: \"3fbb4ca5-d116-4634-a743-1b068b5682ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" Sep 15 12:46:26.162795 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.162760 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb4ca5-d116-4634-a743-1b068b5682ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2\" (UID: \"3fbb4ca5-d116-4634-a743-1b068b5682ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" Sep 15 12:46:26.163408 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.163372 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb4ca5-d116-4634-a743-1b068b5682ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2\" (UID: \"3fbb4ca5-d116-4634-a743-1b068b5682ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" Sep 15 12:46:26.245993 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.245966 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" Sep 15 12:46:26.521616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.521584 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2"] Sep 15 12:46:26.523521 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:46:26.523493 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbb4ca5_d116_4634_a743_1b068b5682ae.slice/crio-c336c0d22d781792aacd4607afd43bfab21046fd849f338ea5bc50a910b7670d WatchSource:0}: Error finding container c336c0d22d781792aacd4607afd43bfab21046fd849f338ea5bc50a910b7670d: Status 404 returned error can't find the container with id c336c0d22d781792aacd4607afd43bfab21046fd849f338ea5bc50a910b7670d Sep 15 12:46:26.910859 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.910822 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" event={"ID":"3fbb4ca5-d116-4634-a743-1b068b5682ae","Type":"ContainerStarted","Data":"cb88277fc378bf61c8a5bab210dc50268519e7a1e8b281117eba47e73dd2f1b5"} Sep 15 12:46:26.911041 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.910880 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" event={"ID":"3fbb4ca5-d116-4634-a743-1b068b5682ae","Type":"ContainerStarted","Data":"c336c0d22d781792aacd4607afd43bfab21046fd849f338ea5bc50a910b7670d"} Sep 15 12:46:26.925861 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:26.925806 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" podStartSLOduration=1.925789634 podStartE2EDuration="1.925789634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:46:26.923754904 +0000 UTC m=+1268.278601230" watchObservedRunningTime="2026-09-15 12:46:26.925789634 +0000 UTC m=+1268.280635957" Sep 15 12:46:27.918166 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:27.918125 2550 generic.go:356] "Generic (PLEG): container finished" podID="3fbb4ca5-d116-4634-a743-1b068b5682ae" containerID="cb88277fc378bf61c8a5bab210dc50268519e7a1e8b281117eba47e73dd2f1b5" exitCode=0 Sep 15 12:46:27.918598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:27.918191 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" event={"ID":"3fbb4ca5-d116-4634-a743-1b068b5682ae","Type":"ContainerDied","Data":"cb88277fc378bf61c8a5bab210dc50268519e7a1e8b281117eba47e73dd2f1b5"} Sep 15 12:46:29.084830 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:29.084806 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" Sep 15 12:46:29.194568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:29.194487 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb4ca5-d116-4634-a743-1b068b5682ae-maas-api-service-ca\") pod \"3fbb4ca5-d116-4634-a743-1b068b5682ae\" (UID: \"3fbb4ca5-d116-4634-a743-1b068b5682ae\") " Sep 15 12:46:29.194900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:29.194873 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbb4ca5-d116-4634-a743-1b068b5682ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fbb4ca5-d116-4634-a743-1b068b5682ae" (UID: "3fbb4ca5-d116-4634-a743-1b068b5682ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:46:29.296476 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:29.296405 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbb4ca5-d116-4634-a743-1b068b5682ae-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:29.932460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:29.932423 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" event={"ID":"3fbb4ca5-d116-4634-a743-1b068b5682ae","Type":"ContainerDied","Data":"c336c0d22d781792aacd4607afd43bfab21046fd849f338ea5bc50a910b7670d"} Sep 15 12:46:29.932460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:29.932448 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2" Sep 15 12:46:29.932668 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:29.932473 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c336c0d22d781792aacd4607afd43bfab21046fd849f338ea5bc50a910b7670d" Sep 15 12:46:36.417840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.417803 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg"] Sep 15 12:46:36.418362 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.418072 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" podUID="0e3a1a2d-221c-474f-b440-923914c51eef" containerName="payload-processing" containerID="cri-o://6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2" gracePeriod=30 Sep 15 12:46:36.442828 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.429846 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc"] Sep 15 12:46:36.442828 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.430166 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" podUID="77304bd3-8e21-4e0f-8004-4b178d077315" containerName="payload-pre-processing" containerID="cri-o://e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc" gracePeriod=30 Sep 15 12:46:36.578324 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.578247 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2"] Sep 15 12:46:36.580724 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.580689 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e3fef-r59t2"] Sep 15 12:46:36.714477 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.714446 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:46:36.744824 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.744797 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:46:36.875505 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.875464 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77304bd3-8e21-4e0f-8004-4b178d077315-plugins-config-volume\") pod \"77304bd3-8e21-4e0f-8004-4b178d077315\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " Sep 15 12:46:36.875703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.875553 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s7p9\" (UniqueName: \"kubernetes.io/projected/0e3a1a2d-221c-474f-b440-923914c51eef-kube-api-access-6s7p9\") pod \"0e3a1a2d-221c-474f-b440-923914c51eef\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " Sep 15 12:46:36.875703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.875582 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tl6n\" (UniqueName: \"kubernetes.io/projected/77304bd3-8e21-4e0f-8004-4b178d077315-kube-api-access-8tl6n\") pod \"77304bd3-8e21-4e0f-8004-4b178d077315\" (UID: \"77304bd3-8e21-4e0f-8004-4b178d077315\") " Sep 15 12:46:36.875703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.875603 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3a1a2d-221c-474f-b440-923914c51eef-plugins-config-volume\") pod \"0e3a1a2d-221c-474f-b440-923914c51eef\" (UID: \"0e3a1a2d-221c-474f-b440-923914c51eef\") " Sep 15 12:46:36.875967 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.875943 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77304bd3-8e21-4e0f-8004-4b178d077315-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77304bd3-8e21-4e0f-8004-4b178d077315" (UID: "77304bd3-8e21-4e0f-8004-4b178d077315"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:46:36.876025 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.876009 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3a1a2d-221c-474f-b440-923914c51eef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e3a1a2d-221c-474f-b440-923914c51eef" (UID: "0e3a1a2d-221c-474f-b440-923914c51eef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:46:36.877809 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.877780 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77304bd3-8e21-4e0f-8004-4b178d077315-kube-api-access-8tl6n" (OuterVolumeSpecName: "kube-api-access-8tl6n") pod "77304bd3-8e21-4e0f-8004-4b178d077315" (UID: "77304bd3-8e21-4e0f-8004-4b178d077315"). InnerVolumeSpecName "kube-api-access-8tl6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:46:36.877809 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.877803 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3a1a2d-221c-474f-b440-923914c51eef-kube-api-access-6s7p9" (OuterVolumeSpecName: "kube-api-access-6s7p9") pod "0e3a1a2d-221c-474f-b440-923914c51eef" (UID: "0e3a1a2d-221c-474f-b440-923914c51eef"). InnerVolumeSpecName "kube-api-access-6s7p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:46:36.974091 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.974004 2550 generic.go:356] "Generic (PLEG): container finished" podID="77304bd3-8e21-4e0f-8004-4b178d077315" containerID="e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc" exitCode=0 Sep 15 12:46:36.974091 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.974045 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" Sep 15 12:46:36.974286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.974096 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" event={"ID":"77304bd3-8e21-4e0f-8004-4b178d077315","Type":"ContainerDied","Data":"e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc"} Sep 15 12:46:36.974286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.974149 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc" event={"ID":"77304bd3-8e21-4e0f-8004-4b178d077315","Type":"ContainerDied","Data":"42d1ef5619186415d034dff6ec4806a730327a46d2063dc1ece0c55de8161944"} Sep 15 12:46:36.974286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.974179 2550 scope.go:117] "RemoveContainer" containerID="e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc" Sep 15 12:46:36.975485 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.975460 2550 generic.go:356] "Generic (PLEG): container finished" podID="0e3a1a2d-221c-474f-b440-923914c51eef" containerID="6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2" exitCode=0 Sep 15 12:46:36.975587 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.975494 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" Sep 15 12:46:36.975587 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.975545 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" event={"ID":"0e3a1a2d-221c-474f-b440-923914c51eef","Type":"ContainerDied","Data":"6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2"} Sep 15 12:46:36.975693 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.975586 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg" event={"ID":"0e3a1a2d-221c-474f-b440-923914c51eef","Type":"ContainerDied","Data":"1c1637a9c78852bed050954f0bcc67039fe24b7a1b19bc3293c15cac8ba71414"} Sep 15 12:46:36.976720 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.976697 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77304bd3-8e21-4e0f-8004-4b178d077315-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:36.976801 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.976726 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s7p9\" (UniqueName: \"kubernetes.io/projected/0e3a1a2d-221c-474f-b440-923914c51eef-kube-api-access-6s7p9\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:36.976801 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.976741 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tl6n\" (UniqueName: \"kubernetes.io/projected/77304bd3-8e21-4e0f-8004-4b178d077315-kube-api-access-8tl6n\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:36.976801 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.976754 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3a1a2d-221c-474f-b440-923914c51eef-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:46:36.985584 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.985563 2550 scope.go:117] "RemoveContainer" containerID="e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc" Sep 15 12:46:36.985918 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:46:36.985893 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc\": container with ID starting with e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc not found: ID does not exist" containerID="e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc" Sep 15 12:46:36.985996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.985929 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc"} err="failed to get container status \"e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc\": rpc error: code = NotFound desc = could not find container \"e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc\": container with ID starting with e7699c52639378c591061d9c2f7141995d365fc58923b0b3c7db5d14fdb9d3fc not found: ID does not exist" Sep 15 12:46:36.985996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.985954 2550 scope.go:117] "RemoveContainer" containerID="6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2" Sep 15 12:46:36.994238 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.994217 2550 scope.go:117] "RemoveContainer" containerID="6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2" Sep 15 12:46:36.994497 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:46:36.994479 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2\": container with ID starting with 6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2 not found: ID does not exist" containerID="6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2" Sep 15 12:46:36.994541 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.994505 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2"} err="failed to get container status \"6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2\": rpc error: code = NotFound desc = could not find container \"6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2\": container with ID starting with 6a968b708d480b7089b33c93a3041183cb0dd93d4b48cfb0ecd58dfdb0fc90b2 not found: ID does not exist" Sep 15 12:46:36.999877 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:36.999856 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc"] Sep 15 12:46:37.003069 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:37.003045 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e3fef-579d575fbb-wfslc"] Sep 15 12:46:37.012609 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:37.012585 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg"] Sep 15 12:46:37.016342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:37.016317 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e3fef-64bdbbb549-676bg"] Sep 15 12:46:37.158260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:37.158230 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e3a1a2d-221c-474f-b440-923914c51eef" path="/var/lib/kubelet/pods/0e3a1a2d-221c-474f-b440-923914c51eef/volumes" Sep 15 12:46:37.158621 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:37.158608 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fbb4ca5-d116-4634-a743-1b068b5682ae" path="/var/lib/kubelet/pods/3fbb4ca5-d116-4634-a743-1b068b5682ae/volumes" Sep 15 12:46:37.158925 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:37.158914 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77304bd3-8e21-4e0f-8004-4b178d077315" path="/var/lib/kubelet/pods/77304bd3-8e21-4e0f-8004-4b178d077315/volumes" Sep 15 12:46:43.730032 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.729995 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp"] Sep 15 12:46:43.730568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730547 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fbb4ca5-d116-4634-a743-1b068b5682ae" containerName="revoke-keys" Sep 15 12:46:43.730623 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730569 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fbb4ca5-d116-4634-a743-1b068b5682ae" containerName="revoke-keys" Sep 15 12:46:43.730623 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730584 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e3a1a2d-221c-474f-b440-923914c51eef" containerName="payload-processing" Sep 15 12:46:43.730623 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730590 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e3a1a2d-221c-474f-b440-923914c51eef" containerName="payload-processing" Sep 15 12:46:43.730623 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730598 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77304bd3-8e21-4e0f-8004-4b178d077315" containerName="payload-pre-processing" Sep 15 12:46:43.730623 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730603 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="77304bd3-8e21-4e0f-8004-4b178d077315" containerName="payload-pre-processing" Sep 15 12:46:43.730781 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730689 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e3a1a2d-221c-474f-b440-923914c51eef" containerName="payload-processing" Sep 15 12:46:43.730781 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730696 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fbb4ca5-d116-4634-a743-1b068b5682ae" containerName="revoke-keys" Sep 15 12:46:43.730781 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.730705 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="77304bd3-8e21-4e0f-8004-4b178d077315" containerName="payload-pre-processing" Sep 15 12:46:43.735148 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.735130 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.737659 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.737638 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-29a9e9-openshift-default-dockercfg-s5ph6\"" Sep 15 12:46:43.744269 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.744243 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp"] Sep 15 12:46:43.842229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.842178 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-socket\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.842666 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.842641 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-certs\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.842917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.842875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-credential-socket\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.842917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.842909 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-envoy\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.843124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.842953 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjv56\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-kube-api-access-wjv56\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.843124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.843014 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-podinfo\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.843124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.843083 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istiod-ca-cert\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.843290 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.843156 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-token\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.843290 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.843218 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-data\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944374 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944341 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istiod-ca-cert\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944559 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944484 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-token\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944559 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944521 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-data\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944559 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944542 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-socket\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944567 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-certs\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944686 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-credential-socket\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944821 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944731 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-envoy\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944821 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944774 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjv56\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-kube-api-access-wjv56\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944958 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944839 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-podinfo\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944958 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944890 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-certs\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.944958 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.944947 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-socket\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.945133 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.945069 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-credential-socket\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.945133 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.945081 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istiod-ca-cert\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.945283 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.945263 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-data\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.947066 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.947042 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-envoy\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.947171 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.947155 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-podinfo\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.952094 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.952069 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-token\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:43.952219 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:43.952202 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjv56\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-kube-api-access-wjv56\") pod \"e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:44.046188 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:44.046155 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:44.323558 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:44.323528 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp"] Sep 15 12:46:44.324557 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:46:44.324529 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c6bf50_16ee_46a7_88ee_8199cf5c3521.slice/crio-e2c59e4cbbeb0db4dc31872d4ba367895a44aaef5eb2d90d260fc45bd8beb5b8 WatchSource:0}: Error finding container e2c59e4cbbeb0db4dc31872d4ba367895a44aaef5eb2d90d260fc45bd8beb5b8: Status 404 returned error can't find the container with id e2c59e4cbbeb0db4dc31872d4ba367895a44aaef5eb2d90d260fc45bd8beb5b8 Sep 15 12:46:44.326855 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:44.326822 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:46:44.326935 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:44.326896 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:46:44.326935 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:44.326924 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:46:45.031394 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:45.031339 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" event={"ID":"46c6bf50-16ee-46a7-88ee-8199cf5c3521","Type":"ContainerStarted","Data":"d63d9afd6ee93d0d0ec93d2d91a60707ae561a41b9ce9ffaaafb145397bbc46f"} Sep 15 12:46:45.031806 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:45.031453 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" event={"ID":"46c6bf50-16ee-46a7-88ee-8199cf5c3521","Type":"ContainerStarted","Data":"e2c59e4cbbeb0db4dc31872d4ba367895a44aaef5eb2d90d260fc45bd8beb5b8"} Sep 15 12:46:45.046891 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:45.046863 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:45.050857 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:45.050810 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podStartSLOduration=2.050796991 podStartE2EDuration="2.050796991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:46:45.049397184 +0000 UTC m=+1286.404243504" watchObservedRunningTime="2026-09-15 12:46:45.050796991 +0000 UTC m=+1286.405643313" Sep 15 12:46:45.052688 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:45.052663 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:46.037098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:46.037067 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:46.038120 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:46.038103 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:46:51.502770 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.502734 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs"] Sep 15 12:46:51.506579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.506556 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:51.508921 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.508899 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-29a9e9-dockercfg-nwzg7\"" Sep 15 12:46:51.509026 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.508927 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-29a9e9\"" Sep 15 12:46:51.516591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.516565 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs"] Sep 15 12:46:51.626900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.626863 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqbb\" (UniqueName: \"kubernetes.io/projected/4e0642b9-7db2-465f-af5a-eadfbb0ce164-kube-api-access-7qqbb\") pod \"payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:51.627086 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.626951 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0642b9-7db2-465f-af5a-eadfbb0ce164-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:51.727930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.727893 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqbb\" (UniqueName: \"kubernetes.io/projected/4e0642b9-7db2-465f-af5a-eadfbb0ce164-kube-api-access-7qqbb\") pod \"payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:51.728109 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.727991 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0642b9-7db2-465f-af5a-eadfbb0ce164-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:51.728730 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.728707 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0642b9-7db2-465f-af5a-eadfbb0ce164-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:51.735255 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.735233 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqbb\" (UniqueName: \"kubernetes.io/projected/4e0642b9-7db2-465f-af5a-eadfbb0ce164-kube-api-access-7qqbb\") pod \"payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:51.819360 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:51.819266 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:52.094516 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:52.094491 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs"] Sep 15 12:46:52.096319 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:46:52.096290 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0642b9_7db2_465f_af5a_eadfbb0ce164.slice/crio-6ed370221f3469fb84150e04852487db5520b54d8b802217b764fc476e939dbf WatchSource:0}: Error finding container 6ed370221f3469fb84150e04852487db5520b54d8b802217b764fc476e939dbf: Status 404 returned error can't find the container with id 6ed370221f3469fb84150e04852487db5520b54d8b802217b764fc476e939dbf Sep 15 12:46:53.080154 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:53.080115 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" event={"ID":"4e0642b9-7db2-465f-af5a-eadfbb0ce164","Type":"ContainerStarted","Data":"9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831"} Sep 15 12:46:53.080614 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:53.080170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" event={"ID":"4e0642b9-7db2-465f-af5a-eadfbb0ce164","Type":"ContainerStarted","Data":"6ed370221f3469fb84150e04852487db5520b54d8b802217b764fc476e939dbf"} Sep 15 12:46:53.080614 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:53.080196 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:46:53.096635 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:46:53.096587 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" podStartSLOduration=2.09657001 podStartE2EDuration="2.09657001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:46:53.095855113 +0000 UTC m=+1294.450701454" watchObservedRunningTime="2026-09-15 12:46:53.09657001 +0000 UTC m=+1294.451416334" Sep 15 12:47:04.088527 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:04.088492 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 12:47:19.338562 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:19.338522 2550 scope.go:117] "RemoveContainer" containerID="eb260612cd9d76d7834995d8f34fadf540ba15000f2487dde132b8f5af729ede" Sep 15 12:47:19.347764 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:19.347740 2550 scope.go:117] "RemoveContainer" containerID="9ba652f8e0827de386b6618f242a0c05d8f9180f2ea643086ca88a5b90ec916e" Sep 15 12:47:19.358785 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:19.358759 2550 scope.go:117] "RemoveContainer" containerID="1515c50a4c074a5f5a43bbd2264e0dc8ca3280fe501e50d9dadfd53dc303e9aa" Sep 15 12:47:22.252996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:22.252960 2550 generic.go:356] "Generic (PLEG): container finished" podID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerID="d63d9afd6ee93d0d0ec93d2d91a60707ae561a41b9ce9ffaaafb145397bbc46f" exitCode=0 Sep 15 12:47:22.253439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:22.253032 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" event={"ID":"46c6bf50-16ee-46a7-88ee-8199cf5c3521","Type":"ContainerDied","Data":"d63d9afd6ee93d0d0ec93d2d91a60707ae561a41b9ce9ffaaafb145397bbc46f"} Sep 15 12:47:22.253533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:22.253518 2550 scope.go:117] "RemoveContainer" containerID="d63d9afd6ee93d0d0ec93d2d91a60707ae561a41b9ce9ffaaafb145397bbc46f" Sep 15 12:47:22.254767 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:22.254735 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:47:22.254884 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:22.254818 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:47:22.254884 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:22.254864 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:47:23.261531 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:23.261485 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" event={"ID":"46c6bf50-16ee-46a7-88ee-8199cf5c3521","Type":"ContainerStarted","Data":"afe390bef12cb9e48496d5898ff993d4d28eafcc01e4af09bbee874064c0c499"} Sep 15 12:47:24.047509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:24.047473 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:47:24.047896 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:24.047804 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 15 12:47:24.048067 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:24.048047 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 15 12:47:25.046625 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.046581 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 15 12:47:25.047124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.046655 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 15 12:47:25.819201 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.819165 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2"] Sep 15 12:47:25.823009 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.822988 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:25.825326 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.825301 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-17f8f8-dockercfg-n9678\"" Sep 15 12:47:25.825796 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.825775 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-17f8f8\"" Sep 15 12:47:25.840399 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.834073 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2"] Sep 15 12:47:25.894977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.894937 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fh8\" (UniqueName: \"kubernetes.io/projected/12f7998d-e46a-4fe8-a744-88818dd86b62-kube-api-access-99fh8\") pod \"payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:25.895348 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.895156 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f7998d-e46a-4fe8-a744-88818dd86b62-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:25.996798 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.996761 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fh8\" (UniqueName: \"kubernetes.io/projected/12f7998d-e46a-4fe8-a744-88818dd86b62-kube-api-access-99fh8\") pod \"payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:25.996982 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.996822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f7998d-e46a-4fe8-a744-88818dd86b62-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:25.997612 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:25.997593 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f7998d-e46a-4fe8-a744-88818dd86b62-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:26.004713 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:26.004686 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fh8\" (UniqueName: \"kubernetes.io/projected/12f7998d-e46a-4fe8-a744-88818dd86b62-kube-api-access-99fh8\") pod \"payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:26.046683 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:26.046632 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 15 12:47:26.047174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:26.046709 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 15 12:47:26.139243 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:26.139142 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:26.424771 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:26.424742 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2"] Sep 15 12:47:26.426630 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:47:26.426595 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f7998d_e46a_4fe8_a744_88818dd86b62.slice/crio-07559b27f2cdc919288e1e585df66858750e3ac129263273455ccb6240931144 WatchSource:0}: Error finding container 07559b27f2cdc919288e1e585df66858750e3ac129263273455ccb6240931144: Status 404 returned error can't find the container with id 07559b27f2cdc919288e1e585df66858750e3ac129263273455ccb6240931144 Sep 15 12:47:27.047367 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:27.046958 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 15 12:47:27.047367 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:27.047021 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 15 12:47:27.288408 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:27.288350 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" event={"ID":"12f7998d-e46a-4fe8-a744-88818dd86b62","Type":"ContainerStarted","Data":"520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242"} Sep 15 12:47:27.288609 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:27.288518 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:47:27.288609 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:27.288540 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" event={"ID":"12f7998d-e46a-4fe8-a744-88818dd86b62","Type":"ContainerStarted","Data":"07559b27f2cdc919288e1e585df66858750e3ac129263273455ccb6240931144"} Sep 15 12:47:27.302783 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:27.302674 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" podStartSLOduration=2.302659615 podStartE2EDuration="2.302659615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:47:27.302319336 +0000 UTC m=+1328.657165659" watchObservedRunningTime="2026-09-15 12:47:27.302659615 +0000 UTC m=+1328.657505937" Sep 15 12:47:28.047077 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:28.047024 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 15 12:47:28.047292 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:28.047102 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 15 12:47:29.046634 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:29.046596 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Sep 15 12:47:29.047015 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:29.046652 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Sep 15 12:47:30.051055 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:30.051025 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:47:30.051437 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:30.051411 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:47:30.052208 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:30.052190 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 12:47:38.296875 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:47:38.296845 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 12:48:17.994122 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:17.994084 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt"] Sep 15 12:48:17.997758 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:17.997741 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" Sep 15 12:48:17.999968 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:17.999947 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:48:18.000084 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:17.999947 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:48:18.005114 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:18.005089 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt"] Sep 15 12:48:18.126339 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:18.126297 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877b5853-058e-4b77-b88b-d4ef3011a259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt\" (UID: \"877b5853-058e-4b77-b88b-d4ef3011a259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" Sep 15 12:48:18.227438 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:18.227372 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877b5853-058e-4b77-b88b-d4ef3011a259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt\" (UID: \"877b5853-058e-4b77-b88b-d4ef3011a259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" Sep 15 12:48:18.227989 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:18.227967 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877b5853-058e-4b77-b88b-d4ef3011a259-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt\" (UID: \"877b5853-058e-4b77-b88b-d4ef3011a259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" Sep 15 12:48:18.311143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:18.311050 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" Sep 15 12:48:18.589037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:18.589006 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt"] Sep 15 12:48:18.591500 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:48:18.591471 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877b5853_058e_4b77_b88b_d4ef3011a259.slice/crio-56c879cac94387661848b86cf5cd8dd687f9693a60ab6e54b715764fb1fb462e WatchSource:0}: Error finding container 56c879cac94387661848b86cf5cd8dd687f9693a60ab6e54b715764fb1fb462e: Status 404 returned error can't find the container with id 56c879cac94387661848b86cf5cd8dd687f9693a60ab6e54b715764fb1fb462e Sep 15 12:48:19.590300 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:19.590262 2550 generic.go:356] "Generic (PLEG): container finished" podID="877b5853-058e-4b77-b88b-d4ef3011a259" containerID="54c5fd3f8ab4bfb99079f22edef55c079706352d325641868dbbff367e98a99f" exitCode=0 Sep 15 12:48:19.590724 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:19.590341 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" event={"ID":"877b5853-058e-4b77-b88b-d4ef3011a259","Type":"ContainerDied","Data":"54c5fd3f8ab4bfb99079f22edef55c079706352d325641868dbbff367e98a99f"} Sep 15 12:48:19.590724 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:19.590419 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" event={"ID":"877b5853-058e-4b77-b88b-d4ef3011a259","Type":"ContainerStarted","Data":"56c879cac94387661848b86cf5cd8dd687f9693a60ab6e54b715764fb1fb462e"} Sep 15 12:48:20.738691 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:20.738663 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" Sep 15 12:48:20.854598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:20.854516 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877b5853-058e-4b77-b88b-d4ef3011a259-maas-api-service-ca\") pod \"877b5853-058e-4b77-b88b-d4ef3011a259\" (UID: \"877b5853-058e-4b77-b88b-d4ef3011a259\") " Sep 15 12:48:20.854880 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:20.854857 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/877b5853-058e-4b77-b88b-d4ef3011a259-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "877b5853-058e-4b77-b88b-d4ef3011a259" (UID: "877b5853-058e-4b77-b88b-d4ef3011a259"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:48:20.955541 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:20.955500 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/877b5853-058e-4b77-b88b-d4ef3011a259-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:48:21.603990 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:21.603962 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" Sep 15 12:48:21.604162 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:21.604004 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt" event={"ID":"877b5853-058e-4b77-b88b-d4ef3011a259","Type":"ContainerDied","Data":"56c879cac94387661848b86cf5cd8dd687f9693a60ab6e54b715764fb1fb462e"} Sep 15 12:48:21.604162 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:21.604049 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c879cac94387661848b86cf5cd8dd687f9693a60ab6e54b715764fb1fb462e" Sep 15 12:48:33.754818 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:33.754772 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9"] Sep 15 12:48:33.755310 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:33.754959 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" podUID="e65aa87c-948a-4d79-86d5-feb5b47b41d0" containerName="payload-processing" containerID="cri-o://ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da" gracePeriod=30 Sep 15 12:48:33.762876 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:33.762847 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks"] Sep 15 12:48:33.763703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:33.763662 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" podUID="a2669a2b-28ba-4856-a745-9785f1edff1b" containerName="payload-pre-processing" containerID="cri-o://85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb" gracePeriod=30 Sep 15 12:48:33.934781 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:33.934596 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt"] Sep 15 12:48:33.938737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:33.938638 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88615e8e-wjhpt"] Sep 15 12:48:34.052233 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.051976 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:48:34.082644 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.082620 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:48:34.182296 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.182254 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65aa87c-948a-4d79-86d5-feb5b47b41d0-plugins-config-volume\") pod \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " Sep 15 12:48:34.182501 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.182366 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2st7\" (UniqueName: \"kubernetes.io/projected/a2669a2b-28ba-4856-a745-9785f1edff1b-kube-api-access-p2st7\") pod \"a2669a2b-28ba-4856-a745-9785f1edff1b\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " Sep 15 12:48:34.182501 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.182464 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2669a2b-28ba-4856-a745-9785f1edff1b-plugins-config-volume\") pod \"a2669a2b-28ba-4856-a745-9785f1edff1b\" (UID: \"a2669a2b-28ba-4856-a745-9785f1edff1b\") " Sep 15 12:48:34.182501 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.182494 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf657\" (UniqueName: \"kubernetes.io/projected/e65aa87c-948a-4d79-86d5-feb5b47b41d0-kube-api-access-tf657\") pod \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\" (UID: \"e65aa87c-948a-4d79-86d5-feb5b47b41d0\") " Sep 15 12:48:34.182856 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.182822 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65aa87c-948a-4d79-86d5-feb5b47b41d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e65aa87c-948a-4d79-86d5-feb5b47b41d0" (UID: "e65aa87c-948a-4d79-86d5-feb5b47b41d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:48:34.182856 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.182849 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2669a2b-28ba-4856-a745-9785f1edff1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2669a2b-28ba-4856-a745-9785f1edff1b" (UID: "a2669a2b-28ba-4856-a745-9785f1edff1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:48:34.184560 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.184538 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65aa87c-948a-4d79-86d5-feb5b47b41d0-kube-api-access-tf657" (OuterVolumeSpecName: "kube-api-access-tf657") pod "e65aa87c-948a-4d79-86d5-feb5b47b41d0" (UID: "e65aa87c-948a-4d79-86d5-feb5b47b41d0"). InnerVolumeSpecName "kube-api-access-tf657". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:48:34.184639 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.184567 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2669a2b-28ba-4856-a745-9785f1edff1b-kube-api-access-p2st7" (OuterVolumeSpecName: "kube-api-access-p2st7") pod "a2669a2b-28ba-4856-a745-9785f1edff1b" (UID: "a2669a2b-28ba-4856-a745-9785f1edff1b"). InnerVolumeSpecName "kube-api-access-p2st7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:48:34.283818 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.283736 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2669a2b-28ba-4856-a745-9785f1edff1b-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:48:34.283818 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.283766 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf657\" (UniqueName: \"kubernetes.io/projected/e65aa87c-948a-4d79-86d5-feb5b47b41d0-kube-api-access-tf657\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:48:34.283818 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.283775 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65aa87c-948a-4d79-86d5-feb5b47b41d0-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:48:34.283818 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.283784 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2st7\" (UniqueName: \"kubernetes.io/projected/a2669a2b-28ba-4856-a745-9785f1edff1b-kube-api-access-p2st7\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:48:34.689008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.688919 2550 generic.go:356] "Generic (PLEG): container finished" podID="e65aa87c-948a-4d79-86d5-feb5b47b41d0" containerID="ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da" exitCode=0 Sep 15 12:48:34.689008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.688966 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" Sep 15 12:48:34.689251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.689004 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" event={"ID":"e65aa87c-948a-4d79-86d5-feb5b47b41d0","Type":"ContainerDied","Data":"ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da"} Sep 15 12:48:34.689251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.689057 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9" event={"ID":"e65aa87c-948a-4d79-86d5-feb5b47b41d0","Type":"ContainerDied","Data":"898c61c0cbe5bae378c546a3899066f5d9181a756e928122a70b4f72f8f73178"} Sep 15 12:48:34.689251 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.689085 2550 scope.go:117] "RemoveContainer" containerID="ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da" Sep 15 12:48:34.690462 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.690436 2550 generic.go:356] "Generic (PLEG): container finished" podID="a2669a2b-28ba-4856-a745-9785f1edff1b" containerID="85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb" exitCode=0 Sep 15 12:48:34.690582 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.690472 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" Sep 15 12:48:34.690582 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.690489 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" event={"ID":"a2669a2b-28ba-4856-a745-9785f1edff1b","Type":"ContainerDied","Data":"85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb"} Sep 15 12:48:34.690582 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.690547 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks" event={"ID":"a2669a2b-28ba-4856-a745-9785f1edff1b","Type":"ContainerDied","Data":"7ce744c0b12526fac748434a817382cece82904512935bed17b489d2da410f5f"} Sep 15 12:48:34.699824 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.699807 2550 scope.go:117] "RemoveContainer" containerID="ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da" Sep 15 12:48:34.700099 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:48:34.700081 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da\": container with ID starting with ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da not found: ID does not exist" containerID="ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da" Sep 15 12:48:34.700156 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.700108 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da"} err="failed to get container status \"ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da\": rpc error: code = NotFound desc = could not find container \"ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da\": container with ID starting with ee3f57d71b8ffa22c528c08877e7ee95678f9bdb4a083b1917734fc32965c0da not found: ID does not exist" Sep 15 12:48:34.700156 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.700126 2550 scope.go:117] "RemoveContainer" containerID="85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb" Sep 15 12:48:34.710761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.710744 2550 scope.go:117] "RemoveContainer" containerID="85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb" Sep 15 12:48:34.711032 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:48:34.711012 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb\": container with ID starting with 85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb not found: ID does not exist" containerID="85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb" Sep 15 12:48:34.711090 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.711037 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb"} err="failed to get container status \"85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb\": rpc error: code = NotFound desc = could not find container \"85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb\": container with ID starting with 85f2f4a9db21bb4f3fa4cb8ba7242404aaeb8a0fb40c3836ae711e14c05240cb not found: ID does not exist" Sep 15 12:48:34.717293 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.717265 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9"] Sep 15 12:48:34.720960 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.720936 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-88615e8e-5694874589-blkp9"] Sep 15 12:48:34.730087 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.730065 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks"] Sep 15 12:48:34.739895 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:34.735908 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-88615e8e-794dcb8878-smfks"] Sep 15 12:48:35.155803 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:35.155769 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="877b5853-058e-4b77-b88b-d4ef3011a259" path="/var/lib/kubelet/pods/877b5853-058e-4b77-b88b-d4ef3011a259/volumes" Sep 15 12:48:35.156168 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:35.156152 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2669a2b-28ba-4856-a745-9785f1edff1b" path="/var/lib/kubelet/pods/a2669a2b-28ba-4856-a745-9785f1edff1b/volumes" Sep 15 12:48:35.156493 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:48:35.156479 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65aa87c-948a-4d79-86d5-feb5b47b41d0" path="/var/lib/kubelet/pods/e65aa87c-948a-4d79-86d5-feb5b47b41d0/volumes" Sep 15 12:50:19.215539 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:19.215511 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:50:19.220039 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:19.220020 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:50:19.426909 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:19.426873 2550 scope.go:117] "RemoveContainer" containerID="48bef63ddf38b10e89def950b78961a8397cf4939cd4e3f28e277247505e5df7" Sep 15 12:50:19.436624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:19.436607 2550 scope.go:117] "RemoveContainer" containerID="62884075e3312e671631a1edc6c50b3969a5c09287b5e2281a62b22df0d40828" Sep 15 12:50:33.542940 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.542903 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f"] Sep 15 12:50:33.543546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543507 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65aa87c-948a-4d79-86d5-feb5b47b41d0" containerName="payload-processing" Sep 15 12:50:33.543546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543525 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65aa87c-948a-4d79-86d5-feb5b47b41d0" containerName="payload-processing" Sep 15 12:50:33.543546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543537 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2669a2b-28ba-4856-a745-9785f1edff1b" containerName="payload-pre-processing" Sep 15 12:50:33.543546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543543 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2669a2b-28ba-4856-a745-9785f1edff1b" containerName="payload-pre-processing" Sep 15 12:50:33.543696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543550 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877b5853-058e-4b77-b88b-d4ef3011a259" containerName="revoke-keys" Sep 15 12:50:33.543696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543556 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="877b5853-058e-4b77-b88b-d4ef3011a259" containerName="revoke-keys" Sep 15 12:50:33.543696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543650 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65aa87c-948a-4d79-86d5-feb5b47b41d0" containerName="payload-processing" Sep 15 12:50:33.543696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543660 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2669a2b-28ba-4856-a745-9785f1edff1b" containerName="payload-pre-processing" Sep 15 12:50:33.543696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.543668 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="877b5853-058e-4b77-b88b-d4ef3011a259" containerName="revoke-keys" Sep 15 12:50:33.548292 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.548275 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.550554 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.550532 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-adc0cb-openshift-default-dockercfg-22prv\"" Sep 15 12:50:33.556014 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.555990 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f"] Sep 15 12:50:33.561477 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561456 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-envoy\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561497 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-socket\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561526 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-credential-socket\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561558 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4psh\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-kube-api-access-q4psh\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561592 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561587 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-token\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561817 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561640 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-data\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561817 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561692 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6593e5a4-ca36-463f-acc3-f9e83a106a93-istiod-ca-cert\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561817 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561724 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-certs\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.561817 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.561751 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-podinfo\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.662825 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.662788 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-envoy\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.662842 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-socket\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.662874 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-credential-socket\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.662911 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4psh\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-kube-api-access-q4psh\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.662942 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-token\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663031 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.662997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-data\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663323 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.663034 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6593e5a4-ca36-463f-acc3-f9e83a106a93-istiod-ca-cert\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663323 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.663070 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-certs\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663323 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.663096 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-podinfo\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.663900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.663874 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-socket\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.664423 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.664260 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6593e5a4-ca36-463f-acc3-f9e83a106a93-istiod-ca-cert\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.664423 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.664356 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-certs\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.664645 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.664503 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-data\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.665017 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.664994 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-credential-socket\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.668689 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.668460 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-podinfo\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.668689 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.668649 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-envoy\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.673092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.673061 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-token\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.674253 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.674204 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4psh\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-kube-api-access-q4psh\") pod \"e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:33.860973 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:33.860888 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:34.135956 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.135927 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f"] Sep 15 12:50:34.136925 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:50:34.136897 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6593e5a4_ca36_463f_acc3_f9e83a106a93.slice/crio-6d0ee6eb8e1f7e2d34ca3391329ca3d7731d9e1bd1f713530207b4fbfdf54ab0 WatchSource:0}: Error finding container 6d0ee6eb8e1f7e2d34ca3391329ca3d7731d9e1bd1f713530207b4fbfdf54ab0: Status 404 returned error can't find the container with id 6d0ee6eb8e1f7e2d34ca3391329ca3d7731d9e1bd1f713530207b4fbfdf54ab0 Sep 15 12:50:34.139818 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.139782 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:50:34.139929 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.139860 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:50:34.139929 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.139903 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:50:34.423703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.423598 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" event={"ID":"6593e5a4-ca36-463f-acc3-f9e83a106a93","Type":"ContainerStarted","Data":"b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a"} Sep 15 12:50:34.423703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.423671 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" event={"ID":"6593e5a4-ca36-463f-acc3-f9e83a106a93","Type":"ContainerStarted","Data":"6d0ee6eb8e1f7e2d34ca3391329ca3d7731d9e1bd1f713530207b4fbfdf54ab0"} Sep 15 12:50:34.448673 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.448601 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podStartSLOduration=1.448579214 podStartE2EDuration="1.448579214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:50:34.441988701 +0000 UTC m=+1515.796835024" watchObservedRunningTime="2026-09-15 12:50:34.448579214 +0000 UTC m=+1515.803425539" Sep 15 12:50:34.861018 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:34.860956 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:35.867163 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:35.867132 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:36.436458 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:36.436414 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:36.437275 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:36.437255 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:50:41.982460 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:41.982372 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt"] Sep 15 12:50:41.986300 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:41.986282 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:41.988622 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:41.988605 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-adc0cb-dockercfg-8vvp5\"" Sep 15 12:50:41.988677 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:41.988639 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-adc0cb\"" Sep 15 12:50:41.996033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:41.996012 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt"] Sep 15 12:50:42.040099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.040068 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c82fea-ddc9-49bc-a25b-0662ca14f606-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:42.040286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.040195 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxcgw\" (UniqueName: \"kubernetes.io/projected/93c82fea-ddc9-49bc-a25b-0662ca14f606-kube-api-access-sxcgw\") pod \"payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:42.141564 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.141529 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c82fea-ddc9-49bc-a25b-0662ca14f606-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:42.141752 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.141623 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxcgw\" (UniqueName: \"kubernetes.io/projected/93c82fea-ddc9-49bc-a25b-0662ca14f606-kube-api-access-sxcgw\") pod \"payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:42.142230 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.142207 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c82fea-ddc9-49bc-a25b-0662ca14f606-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:42.149394 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.149350 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxcgw\" (UniqueName: \"kubernetes.io/projected/93c82fea-ddc9-49bc-a25b-0662ca14f606-kube-api-access-sxcgw\") pod \"payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:42.299076 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.299038 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:42.570446 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:42.570422 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt"] Sep 15 12:50:42.572076 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:50:42.572052 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c82fea_ddc9_49bc_a25b_0662ca14f606.slice/crio-b6491fa8da9f9a7fff1791ddea15cc8b450e2b893cb93c734b0f4c93bdfd4daf WatchSource:0}: Error finding container b6491fa8da9f9a7fff1791ddea15cc8b450e2b893cb93c734b0f4c93bdfd4daf: Status 404 returned error can't find the container with id b6491fa8da9f9a7fff1791ddea15cc8b450e2b893cb93c734b0f4c93bdfd4daf Sep 15 12:50:43.478351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:43.478305 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" event={"ID":"93c82fea-ddc9-49bc-a25b-0662ca14f606","Type":"ContainerStarted","Data":"40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961"} Sep 15 12:50:43.478742 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:43.478365 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" event={"ID":"93c82fea-ddc9-49bc-a25b-0662ca14f606","Type":"ContainerStarted","Data":"b6491fa8da9f9a7fff1791ddea15cc8b450e2b893cb93c734b0f4c93bdfd4daf"} Sep 15 12:50:43.478742 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:43.478405 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:50:43.493671 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:43.493618 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" podStartSLOduration=2.493603333 podStartE2EDuration="2.493603333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:50:43.49240854 +0000 UTC m=+1524.847254856" watchObservedRunningTime="2026-09-15 12:50:43.493603333 +0000 UTC m=+1524.848449655" Sep 15 12:50:54.487867 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:50:54.487828 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:51:02.613754 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:02.613721 2550 generic.go:356] "Generic (PLEG): container finished" podID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerID="b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a" exitCode=0 Sep 15 12:51:02.614166 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:02.613799 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" event={"ID":"6593e5a4-ca36-463f-acc3-f9e83a106a93","Type":"ContainerDied","Data":"b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a"} Sep 15 12:51:02.614227 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:02.614211 2550 scope.go:117] "RemoveContainer" containerID="b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a" Sep 15 12:51:02.615365 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:02.615336 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:51:02.615452 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:02.615437 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:51:02.615494 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:02.615472 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:51:03.621583 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:03.621549 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" event={"ID":"6593e5a4-ca36-463f-acc3-f9e83a106a93","Type":"ContainerStarted","Data":"e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5"} Sep 15 12:51:03.861874 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:03.861837 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:51:03.863317 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:03.863287 2550 patch_prober.go:28] interesting pod/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 15 12:51:03.863507 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:03.863338 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 15 12:51:04.862139 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:04.862090 2550 patch_prober.go:28] interesting pod/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 15 12:51:04.862687 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:04.862166 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 15 12:51:05.862037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:05.862000 2550 patch_prober.go:28] interesting pod/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 15 12:51:05.862222 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:05.862058 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 15 12:51:06.437404 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:06.436514 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:51:06.861520 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:06.861485 2550 patch_prober.go:28] interesting pod/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 15 12:51:06.861703 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:06.861546 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 15 12:51:07.861595 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:07.861555 2550 patch_prober.go:28] interesting pod/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 15 12:51:07.862091 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:07.861621 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 15 12:51:08.861312 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:08.861264 2550 patch_prober.go:28] interesting pod/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 15 12:51:08.861514 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:08.861323 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 15 12:51:09.865939 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:09.865908 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:51:09.867155 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:09.867135 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:51:15.939108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.939068 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79"] Sep 15 12:51:15.944008 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.943990 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:15.946410 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.946375 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-25e7f8-dockercfg-lwdff\"" Sep 15 12:51:15.946507 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.946414 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-25e7f8\"" Sep 15 12:51:15.955526 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.955497 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79"] Sep 15 12:51:15.980453 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.980413 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw"] Sep 15 12:51:15.984063 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.984043 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:15.992705 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:15.992677 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw"] Sep 15 12:51:16.000107 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.000072 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9g8s\" (UniqueName: \"kubernetes.io/projected/f55c1e87-a5e9-4609-a382-38af66d46b7a-kube-api-access-s9g8s\") pod \"payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.000342 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.000312 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-plugins-config-volume\") pod \"payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.000524 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.000491 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572dk\" (UniqueName: \"kubernetes.io/projected/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-kube-api-access-572dk\") pod \"payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.000605 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.000588 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55c1e87-a5e9-4609-a382-38af66d46b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.101471 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.101432 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55c1e87-a5e9-4609-a382-38af66d46b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.101648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.101485 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9g8s\" (UniqueName: \"kubernetes.io/projected/f55c1e87-a5e9-4609-a382-38af66d46b7a-kube-api-access-s9g8s\") pod \"payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.101648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.101520 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-plugins-config-volume\") pod \"payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.101648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.101598 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572dk\" (UniqueName: \"kubernetes.io/projected/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-kube-api-access-572dk\") pod \"payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.102152 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.102133 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55c1e87-a5e9-4609-a382-38af66d46b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.102364 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.102344 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-plugins-config-volume\") pod \"payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.109840 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.109816 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572dk\" (UniqueName: \"kubernetes.io/projected/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-kube-api-access-572dk\") pod \"payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.109959 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.109889 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9g8s\" (UniqueName: \"kubernetes.io/projected/f55c1e87-a5e9-4609-a382-38af66d46b7a-kube-api-access-s9g8s\") pod \"payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.263890 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.263853 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.296162 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.296136 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.542447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.542417 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79"] Sep 15 12:51:16.544567 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:51:16.544538 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55c1e87_a5e9_4609_a382_38af66d46b7a.slice/crio-6804a95c2936697f912b280234c8562c3c53fe55bab91981d96a20dc710121d9 WatchSource:0}: Error finding container 6804a95c2936697f912b280234c8562c3c53fe55bab91981d96a20dc710121d9: Status 404 returned error can't find the container with id 6804a95c2936697f912b280234c8562c3c53fe55bab91981d96a20dc710121d9 Sep 15 12:51:16.590968 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.590945 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw"] Sep 15 12:51:16.592473 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:51:16.592453 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e056ffc_0e74_4f74_9da7_f5bbb9330b69.slice/crio-363988969f8421cca76c4744bf25b20a56575c1fd0076fdcff9587011fcd8c16 WatchSource:0}: Error finding container 363988969f8421cca76c4744bf25b20a56575c1fd0076fdcff9587011fcd8c16: Status 404 returned error can't find the container with id 363988969f8421cca76c4744bf25b20a56575c1fd0076fdcff9587011fcd8c16 Sep 15 12:51:16.709150 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.709113 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" event={"ID":"f55c1e87-a5e9-4609-a382-38af66d46b7a","Type":"ContainerStarted","Data":"f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64"} Sep 15 12:51:16.709506 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.709185 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:16.709506 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.709212 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" event={"ID":"f55c1e87-a5e9-4609-a382-38af66d46b7a","Type":"ContainerStarted","Data":"6804a95c2936697f912b280234c8562c3c53fe55bab91981d96a20dc710121d9"} Sep 15 12:51:16.710827 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.710803 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" event={"ID":"8e056ffc-0e74-4f74-9da7-f5bbb9330b69","Type":"ContainerStarted","Data":"e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659"} Sep 15 12:51:16.710916 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.710844 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" event={"ID":"8e056ffc-0e74-4f74-9da7-f5bbb9330b69","Type":"ContainerStarted","Data":"363988969f8421cca76c4744bf25b20a56575c1fd0076fdcff9587011fcd8c16"} Sep 15 12:51:16.710916 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.710864 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:16.725828 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.725772 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" podStartSLOduration=1.7257568540000001 podStartE2EDuration="1.725756854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:51:16.724585398 +0000 UTC m=+1558.079431747" watchObservedRunningTime="2026-09-15 12:51:16.725756854 +0000 UTC m=+1558.080603176" Sep 15 12:51:16.741846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:16.741794 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" podStartSLOduration=1.741778636 podStartE2EDuration="1.741778636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:51:16.739726714 +0000 UTC m=+1558.094573047" watchObservedRunningTime="2026-09-15 12:51:16.741778636 +0000 UTC m=+1558.096625012" Sep 15 12:51:19.465897 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:19.465861 2550 scope.go:117] "RemoveContainer" containerID="4c0a1d8df027382138a28f361d09b23b89e9be835616382bc2a72a98c651b799" Sep 15 12:51:27.719811 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:27.719774 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:51:27.720263 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:27.719850 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:51:42.838064 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:42.838023 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx"] Sep 15 12:51:42.844341 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:42.844306 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" Sep 15 12:51:42.848155 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:42.848124 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx"] Sep 15 12:51:42.848841 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:42.848816 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:51:42.849179 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:42.848765 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:51:42.988442 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:42.988404 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18022aa-d335-4029-92d3-b723ef76b71e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx\" (UID: \"e18022aa-d335-4029-92d3-b723ef76b71e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" Sep 15 12:51:43.089531 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:43.089424 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18022aa-d335-4029-92d3-b723ef76b71e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx\" (UID: \"e18022aa-d335-4029-92d3-b723ef76b71e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" Sep 15 12:51:43.090087 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:43.090061 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18022aa-d335-4029-92d3-b723ef76b71e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx\" (UID: \"e18022aa-d335-4029-92d3-b723ef76b71e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" Sep 15 12:51:43.158977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:43.158948 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" Sep 15 12:51:43.429101 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:43.429070 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx"] Sep 15 12:51:43.438886 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:51:43.430784 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18022aa_d335_4029_92d3_b723ef76b71e.slice/crio-1eb1f27574f471e9e9d3632ea0d9dc51309fcc8033e61cac323cdc01bc60535f WatchSource:0}: Error finding container 1eb1f27574f471e9e9d3632ea0d9dc51309fcc8033e61cac323cdc01bc60535f: Status 404 returned error can't find the container with id 1eb1f27574f471e9e9d3632ea0d9dc51309fcc8033e61cac323cdc01bc60535f Sep 15 12:51:43.885161 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:43.885126 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" event={"ID":"e18022aa-d335-4029-92d3-b723ef76b71e","Type":"ContainerStarted","Data":"ebfcbad75f7e37ca936aea28809b87afa2ca48cc3e96ac3f4060e05b6a5876a2"} Sep 15 12:51:43.885572 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:43.885216 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" event={"ID":"e18022aa-d335-4029-92d3-b723ef76b71e","Type":"ContainerStarted","Data":"1eb1f27574f471e9e9d3632ea0d9dc51309fcc8033e61cac323cdc01bc60535f"} Sep 15 12:51:43.900177 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:43.900122 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" podStartSLOduration=1.9001061209999999 podStartE2EDuration="1.900106121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:51:43.898324199 +0000 UTC m=+1585.253170522" watchObservedRunningTime="2026-09-15 12:51:43.900106121 +0000 UTC m=+1585.254952445" Sep 15 12:51:44.892253 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:44.892215 2550 generic.go:356] "Generic (PLEG): container finished" podID="e18022aa-d335-4029-92d3-b723ef76b71e" containerID="ebfcbad75f7e37ca936aea28809b87afa2ca48cc3e96ac3f4060e05b6a5876a2" exitCode=0 Sep 15 12:51:44.892736 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:44.892254 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" event={"ID":"e18022aa-d335-4029-92d3-b723ef76b71e","Type":"ContainerDied","Data":"ebfcbad75f7e37ca936aea28809b87afa2ca48cc3e96ac3f4060e05b6a5876a2"} Sep 15 12:51:46.041709 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:46.041685 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" Sep 15 12:51:46.218850 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:46.218764 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18022aa-d335-4029-92d3-b723ef76b71e-maas-api-service-ca\") pod \"e18022aa-d335-4029-92d3-b723ef76b71e\" (UID: \"e18022aa-d335-4029-92d3-b723ef76b71e\") " Sep 15 12:51:46.219138 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:46.219114 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18022aa-d335-4029-92d3-b723ef76b71e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e18022aa-d335-4029-92d3-b723ef76b71e" (UID: "e18022aa-d335-4029-92d3-b723ef76b71e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:51:46.319793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:46.319763 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18022aa-d335-4029-92d3-b723ef76b71e-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:51:46.904728 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:46.904696 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" Sep 15 12:51:46.904901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:46.904728 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx" event={"ID":"e18022aa-d335-4029-92d3-b723ef76b71e","Type":"ContainerDied","Data":"1eb1f27574f471e9e9d3632ea0d9dc51309fcc8033e61cac323cdc01bc60535f"} Sep 15 12:51:46.904901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:46.904774 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eb1f27574f471e9e9d3632ea0d9dc51309fcc8033e61cac323cdc01bc60535f" Sep 15 12:51:58.705583 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.705545 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt"] Sep 15 12:51:58.705988 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.705816 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" podUID="93c82fea-ddc9-49bc-a25b-0662ca14f606" containerName="payload-pre-processing" containerID="cri-o://40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961" gracePeriod=30 Sep 15 12:51:58.888960 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.888545 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx"] Sep 15 12:51:58.891492 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.891461 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adc0cb-j78lx"] Sep 15 12:51:58.969678 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.969620 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:51:58.978350 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.978321 2550 generic.go:356] "Generic (PLEG): container finished" podID="93c82fea-ddc9-49bc-a25b-0662ca14f606" containerID="40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961" exitCode=0 Sep 15 12:51:58.978495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.978368 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" Sep 15 12:51:58.978495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.978410 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" event={"ID":"93c82fea-ddc9-49bc-a25b-0662ca14f606","Type":"ContainerDied","Data":"40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961"} Sep 15 12:51:58.978495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.978461 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt" event={"ID":"93c82fea-ddc9-49bc-a25b-0662ca14f606","Type":"ContainerDied","Data":"b6491fa8da9f9a7fff1791ddea15cc8b450e2b893cb93c734b0f4c93bdfd4daf"} Sep 15 12:51:58.978495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.978489 2550 scope.go:117] "RemoveContainer" containerID="40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961" Sep 15 12:51:58.988497 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.988476 2550 scope.go:117] "RemoveContainer" containerID="40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961" Sep 15 12:51:58.988853 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:51:58.988830 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961\": container with ID starting with 40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961 not found: ID does not exist" containerID="40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961" Sep 15 12:51:58.988921 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:58.988864 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961"} err="failed to get container status \"40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961\": rpc error: code = NotFound desc = could not find container \"40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961\": container with ID starting with 40a20f5d102875f89e41057c08b1ddd101e6629b24349f35a12644026c5b8961 not found: ID does not exist" Sep 15 12:51:59.038228 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.038194 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxcgw\" (UniqueName: \"kubernetes.io/projected/93c82fea-ddc9-49bc-a25b-0662ca14f606-kube-api-access-sxcgw\") pod \"93c82fea-ddc9-49bc-a25b-0662ca14f606\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " Sep 15 12:51:59.038456 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.038365 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c82fea-ddc9-49bc-a25b-0662ca14f606-plugins-config-volume\") pod \"93c82fea-ddc9-49bc-a25b-0662ca14f606\" (UID: \"93c82fea-ddc9-49bc-a25b-0662ca14f606\") " Sep 15 12:51:59.039119 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.039085 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c82fea-ddc9-49bc-a25b-0662ca14f606-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93c82fea-ddc9-49bc-a25b-0662ca14f606" (UID: "93c82fea-ddc9-49bc-a25b-0662ca14f606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:51:59.042142 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.042112 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c82fea-ddc9-49bc-a25b-0662ca14f606-kube-api-access-sxcgw" (OuterVolumeSpecName: "kube-api-access-sxcgw") pod "93c82fea-ddc9-49bc-a25b-0662ca14f606" (UID: "93c82fea-ddc9-49bc-a25b-0662ca14f606"). InnerVolumeSpecName "kube-api-access-sxcgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:51:59.139837 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.139791 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxcgw\" (UniqueName: \"kubernetes.io/projected/93c82fea-ddc9-49bc-a25b-0662ca14f606-kube-api-access-sxcgw\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:51:59.139837 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.139830 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c82fea-ddc9-49bc-a25b-0662ca14f606-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:51:59.156631 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.156588 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e18022aa-d335-4029-92d3-b723ef76b71e" path="/var/lib/kubelet/pods/e18022aa-d335-4029-92d3-b723ef76b71e/volumes" Sep 15 12:51:59.295203 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.295163 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt"] Sep 15 12:51:59.297598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:51:59.297575 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-adc0cb-5ffbbc9f66-rkvxt"] Sep 15 12:52:01.157092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:01.157059 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93c82fea-ddc9-49bc-a25b-0662ca14f606" path="/var/lib/kubelet/pods/93c82fea-ddc9-49bc-a25b-0662ca14f606/volumes" Sep 15 12:52:05.431923 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.431884 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f"] Sep 15 12:52:05.432309 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.432165 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" containerID="cri-o://e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5" gracePeriod=30 Sep 15 12:52:05.794214 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.794180 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8"] Sep 15 12:52:05.794688 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.794675 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93c82fea-ddc9-49bc-a25b-0662ca14f606" containerName="payload-pre-processing" Sep 15 12:52:05.794737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.794690 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="93c82fea-ddc9-49bc-a25b-0662ca14f606" containerName="payload-pre-processing" Sep 15 12:52:05.794737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.794709 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e18022aa-d335-4029-92d3-b723ef76b71e" containerName="revoke-keys" Sep 15 12:52:05.794737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.794715 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e18022aa-d335-4029-92d3-b723ef76b71e" containerName="revoke-keys" Sep 15 12:52:05.794836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.794805 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e18022aa-d335-4029-92d3-b723ef76b71e" containerName="revoke-keys" Sep 15 12:52:05.794836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.794812 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="93c82fea-ddc9-49bc-a25b-0662ca14f606" containerName="payload-pre-processing" Sep 15 12:52:05.799117 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.799100 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" Sep 15 12:52:05.801632 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.801599 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:52:05.802490 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.802464 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:52:05.804304 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.804281 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8"] Sep 15 12:52:05.910620 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:05.910585 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8\" (UID: \"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" Sep 15 12:52:06.011482 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:06.011445 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8\" (UID: \"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" Sep 15 12:52:06.012109 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:06.012081 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8\" (UID: \"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" Sep 15 12:52:06.113650 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:06.113554 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" Sep 15 12:52:06.383135 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:06.383106 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8"] Sep 15 12:52:06.385810 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:52:06.385786 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37560bc8_3bdf_4ce9_8f13_dcad8f03e6ae.slice/crio-3512a3ae8a26ec69f407a76c796e78745af0c4931ae2bdd2c2772afcd6c472c1 WatchSource:0}: Error finding container 3512a3ae8a26ec69f407a76c796e78745af0c4931ae2bdd2c2772afcd6c472c1: Status 404 returned error can't find the container with id 3512a3ae8a26ec69f407a76c796e78745af0c4931ae2bdd2c2772afcd6c472c1 Sep 15 12:52:07.034934 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:07.034896 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" event={"ID":"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae","Type":"ContainerStarted","Data":"eacddb07d11780718065ef8b726f6dc86ebadcb87213f8c69701eb5c42d866e3"} Sep 15 12:52:07.035318 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:07.034949 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" event={"ID":"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae","Type":"ContainerStarted","Data":"3512a3ae8a26ec69f407a76c796e78745af0c4931ae2bdd2c2772afcd6c472c1"} Sep 15 12:52:07.051216 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:07.051169 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" podStartSLOduration=2.051140693 podStartE2EDuration="2.051140693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:52:07.049343439 +0000 UTC m=+1608.404189764" watchObservedRunningTime="2026-09-15 12:52:07.051140693 +0000 UTC m=+1608.405987017" Sep 15 12:52:08.042038 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:08.041997 2550 generic.go:356] "Generic (PLEG): container finished" podID="37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae" containerID="eacddb07d11780718065ef8b726f6dc86ebadcb87213f8c69701eb5c42d866e3" exitCode=0 Sep 15 12:52:08.042441 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:08.042079 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" event={"ID":"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae","Type":"ContainerDied","Data":"eacddb07d11780718065ef8b726f6dc86ebadcb87213f8c69701eb5c42d866e3"} Sep 15 12:52:09.192343 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:09.192313 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" Sep 15 12:52:09.246884 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:09.246850 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae-maas-api-service-ca\") pod \"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae\" (UID: \"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae\") " Sep 15 12:52:09.247189 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:09.247167 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae" (UID: "37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:09.348451 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:09.348348 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:09.868835 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:09.868797 2550 patch_prober.go:28] interesting pod/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 12:52:09.869006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:09.868855 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 12:52:10.055766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.055737 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" Sep 15 12:52:10.055766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.055751 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8" event={"ID":"37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae","Type":"ContainerDied","Data":"3512a3ae8a26ec69f407a76c796e78745af0c4931ae2bdd2c2772afcd6c472c1"} Sep 15 12:52:10.055966 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.055799 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3512a3ae8a26ec69f407a76c796e78745af0c4931ae2bdd2c2772afcd6c472c1" Sep 15 12:52:10.701788 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.701732 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:52:10.762186 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762157 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-credential-socket\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762208 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-token\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762230 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6593e5a4-ca36-463f-acc3-f9e83a106a93-istiod-ca-cert\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762267 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-certs\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762289 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-envoy\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762537 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762448 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4psh\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-kube-api-access-q4psh\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762537 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762505 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-podinfo\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762537 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762509 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:10.762661 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762544 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-socket\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762661 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762587 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:10.762661 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762652 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-data\") pod \"6593e5a4-ca36-463f-acc3-f9e83a106a93\" (UID: \"6593e5a4-ca36-463f-acc3-f9e83a106a93\") " Sep 15 12:52:10.762932 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762815 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6593e5a4-ca36-463f-acc3-f9e83a106a93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:10.763061 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.762975 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:10.763161 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.763129 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-data" (OuterVolumeSpecName: "istio-data") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:10.763280 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.763213 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.763280 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.763232 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6593e5a4-ca36-463f-acc3-f9e83a106a93-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.763280 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.763241 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.763280 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.763249 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.763280 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.763257 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.764674 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.764646 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:10.764803 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.764781 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 12:52:10.764931 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.764910 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-kube-api-access-q4psh" (OuterVolumeSpecName: "kube-api-access-q4psh") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "kube-api-access-q4psh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:10.764981 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.764939 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-token" (OuterVolumeSpecName: "istio-token") pod "6593e5a4-ca36-463f-acc3-f9e83a106a93" (UID: "6593e5a4-ca36-463f-acc3-f9e83a106a93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:10.864661 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.864625 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.864661 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.864655 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.864661 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.864669 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4psh\" (UniqueName: \"kubernetes.io/projected/6593e5a4-ca36-463f-acc3-f9e83a106a93-kube-api-access-q4psh\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:10.864898 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:10.864682 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6593e5a4-ca36-463f-acc3-f9e83a106a93-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:11.067011 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.066974 2550 generic.go:356] "Generic (PLEG): container finished" podID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerID="e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5" exitCode=0 Sep 15 12:52:11.067181 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.067022 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" Sep 15 12:52:11.067181 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.067051 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" event={"ID":"6593e5a4-ca36-463f-acc3-f9e83a106a93","Type":"ContainerDied","Data":"e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5"} Sep 15 12:52:11.067181 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.067108 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f" event={"ID":"6593e5a4-ca36-463f-acc3-f9e83a106a93","Type":"ContainerDied","Data":"6d0ee6eb8e1f7e2d34ca3391329ca3d7731d9e1bd1f713530207b4fbfdf54ab0"} Sep 15 12:52:11.067181 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.067136 2550 scope.go:117] "RemoveContainer" containerID="e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5" Sep 15 12:52:11.080770 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.080750 2550 scope.go:117] "RemoveContainer" containerID="b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a" Sep 15 12:52:11.090621 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.090484 2550 scope.go:117] "RemoveContainer" containerID="e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5" Sep 15 12:52:11.090973 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:11.090954 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5\": container with ID starting with e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5 not found: ID does not exist" containerID="e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5" Sep 15 12:52:11.091042 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.090982 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5"} err="failed to get container status \"e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5\": rpc error: code = NotFound desc = could not find container \"e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5\": container with ID starting with e50c95d958f1f76805575e32e70d53d1e806a46720d750dfdbc059e5c03e2df5 not found: ID does not exist" Sep 15 12:52:11.091042 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.091002 2550 scope.go:117] "RemoveContainer" containerID="b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a" Sep 15 12:52:11.091299 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:11.091282 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a\": container with ID starting with b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a not found: ID does not exist" containerID="b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a" Sep 15 12:52:11.091351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.091303 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a"} err="failed to get container status \"b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a\": rpc error: code = NotFound desc = could not find container \"b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a\": container with ID starting with b4aa7d40d8cdc879e35ccbf3b36fc6ffd2c868e4ea2748a4934c4aece15abc3a not found: ID does not exist" Sep 15 12:52:11.093217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.093196 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f"] Sep 15 12:52:11.096998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.096977 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-adc0cb-openshift-default-87cbc6c4-t965f"] Sep 15 12:52:11.155829 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:11.155747 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" path="/var/lib/kubelet/pods/6593e5a4-ca36-463f-acc3-f9e83a106a93/volumes" Sep 15 12:52:19.487839 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:19.487810 2550 scope.go:117] "RemoveContainer" containerID="a2c9fd1048a5523d667e53606cbea372a25be04cdd0853318ea2ca9da5711abc" Sep 15 12:52:21.660236 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.660199 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw"] Sep 15 12:52:21.660721 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.660507 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" podUID="8e056ffc-0e74-4f74-9da7-f5bbb9330b69" containerName="payload-processing" containerID="cri-o://e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659" gracePeriod=30 Sep 15 12:52:21.667543 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.667464 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79"] Sep 15 12:52:21.667709 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.667683 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" podUID="f55c1e87-a5e9-4609-a382-38af66d46b7a" containerName="payload-pre-processing" containerID="cri-o://f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64" gracePeriod=30 Sep 15 12:52:21.832177 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.832116 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8"] Sep 15 12:52:21.836459 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.836435 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-25e7f8-6lpk8"] Sep 15 12:52:21.920257 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.920199 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:52:21.965108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:21.965084 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:52:22.078516 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.078483 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-572dk\" (UniqueName: \"kubernetes.io/projected/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-kube-api-access-572dk\") pod \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " Sep 15 12:52:22.078705 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.078590 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55c1e87-a5e9-4609-a382-38af66d46b7a-plugins-config-volume\") pod \"f55c1e87-a5e9-4609-a382-38af66d46b7a\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " Sep 15 12:52:22.078705 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.078642 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-plugins-config-volume\") pod \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\" (UID: \"8e056ffc-0e74-4f74-9da7-f5bbb9330b69\") " Sep 15 12:52:22.078835 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.078725 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9g8s\" (UniqueName: \"kubernetes.io/projected/f55c1e87-a5e9-4609-a382-38af66d46b7a-kube-api-access-s9g8s\") pod \"f55c1e87-a5e9-4609-a382-38af66d46b7a\" (UID: \"f55c1e87-a5e9-4609-a382-38af66d46b7a\") " Sep 15 12:52:22.079021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.078996 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55c1e87-a5e9-4609-a382-38af66d46b7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f55c1e87-a5e9-4609-a382-38af66d46b7a" (UID: "f55c1e87-a5e9-4609-a382-38af66d46b7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:22.079100 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.079075 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e056ffc-0e74-4f74-9da7-f5bbb9330b69" (UID: "8e056ffc-0e74-4f74-9da7-f5bbb9330b69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:22.080721 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.080697 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-kube-api-access-572dk" (OuterVolumeSpecName: "kube-api-access-572dk") pod "8e056ffc-0e74-4f74-9da7-f5bbb9330b69" (UID: "8e056ffc-0e74-4f74-9da7-f5bbb9330b69"). InnerVolumeSpecName "kube-api-access-572dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:22.080775 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.080705 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55c1e87-a5e9-4609-a382-38af66d46b7a-kube-api-access-s9g8s" (OuterVolumeSpecName: "kube-api-access-s9g8s") pod "f55c1e87-a5e9-4609-a382-38af66d46b7a" (UID: "f55c1e87-a5e9-4609-a382-38af66d46b7a"). InnerVolumeSpecName "kube-api-access-s9g8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:22.135169 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.135136 2550 generic.go:356] "Generic (PLEG): container finished" podID="f55c1e87-a5e9-4609-a382-38af66d46b7a" containerID="f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64" exitCode=0 Sep 15 12:52:22.135306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.135178 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" Sep 15 12:52:22.135306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.135226 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" event={"ID":"f55c1e87-a5e9-4609-a382-38af66d46b7a","Type":"ContainerDied","Data":"f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64"} Sep 15 12:52:22.135306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.135274 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79" event={"ID":"f55c1e87-a5e9-4609-a382-38af66d46b7a","Type":"ContainerDied","Data":"6804a95c2936697f912b280234c8562c3c53fe55bab91981d96a20dc710121d9"} Sep 15 12:52:22.135306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.135298 2550 scope.go:117] "RemoveContainer" containerID="f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64" Sep 15 12:52:22.136667 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.136643 2550 generic.go:356] "Generic (PLEG): container finished" podID="8e056ffc-0e74-4f74-9da7-f5bbb9330b69" containerID="e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659" exitCode=0 Sep 15 12:52:22.136793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.136668 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" event={"ID":"8e056ffc-0e74-4f74-9da7-f5bbb9330b69","Type":"ContainerDied","Data":"e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659"} Sep 15 12:52:22.136793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.136713 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" event={"ID":"8e056ffc-0e74-4f74-9da7-f5bbb9330b69","Type":"ContainerDied","Data":"363988969f8421cca76c4744bf25b20a56575c1fd0076fdcff9587011fcd8c16"} Sep 15 12:52:22.136793 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.136676 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw" Sep 15 12:52:22.144723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.144650 2550 scope.go:117] "RemoveContainer" containerID="f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64" Sep 15 12:52:22.145408 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:22.145194 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64\": container with ID starting with f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64 not found: ID does not exist" containerID="f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64" Sep 15 12:52:22.145408 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.145230 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64"} err="failed to get container status \"f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64\": rpc error: code = NotFound desc = could not find container \"f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64\": container with ID starting with f0f1f62b22ee8cc6b7d2e8dd4ba6bba3fee2c8f198c0721e70f09a60203f4a64 not found: ID does not exist" Sep 15 12:52:22.145408 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.145252 2550 scope.go:117] "RemoveContainer" containerID="e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659" Sep 15 12:52:22.154102 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.154085 2550 scope.go:117] "RemoveContainer" containerID="e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659" Sep 15 12:52:22.154350 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:22.154333 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659\": container with ID starting with e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659 not found: ID does not exist" containerID="e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659" Sep 15 12:52:22.154426 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.154357 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659"} err="failed to get container status \"e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659\": rpc error: code = NotFound desc = could not find container \"e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659\": container with ID starting with e8c447243a63286145cf5f40f6d6ebbaa163fb2c4d6aa726fec090e91bef4659 not found: ID does not exist" Sep 15 12:52:22.167500 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.167474 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79"] Sep 15 12:52:22.169575 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.169555 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-25e7f8-644867cb6b-d8w79"] Sep 15 12:52:22.175319 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.175264 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw"] Sep 15 12:52:22.179595 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.179573 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55c1e87-a5e9-4609-a382-38af66d46b7a-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:22.179595 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.179594 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:22.179714 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.179605 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9g8s\" (UniqueName: \"kubernetes.io/projected/f55c1e87-a5e9-4609-a382-38af66d46b7a-kube-api-access-s9g8s\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:22.179714 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.179619 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572dk\" (UniqueName: \"kubernetes.io/projected/8e056ffc-0e74-4f74-9da7-f5bbb9330b69-kube-api-access-572dk\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:22.181222 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:22.181203 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-25e7f8-5f89fcb4c9-dhtnw"] Sep 15 12:52:23.154366 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:23.154332 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae" path="/var/lib/kubelet/pods/37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae/volumes" Sep 15 12:52:23.154768 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:23.154752 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e056ffc-0e74-4f74-9da7-f5bbb9330b69" path="/var/lib/kubelet/pods/8e056ffc-0e74-4f74-9da7-f5bbb9330b69/volumes" Sep 15 12:52:23.155090 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:23.155078 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55c1e87-a5e9-4609-a382-38af66d46b7a" path="/var/lib/kubelet/pods/f55c1e87-a5e9-4609-a382-38af66d46b7a/volumes" Sep 15 12:52:28.449100 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449062 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws"] Sep 15 12:52:28.449545 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449531 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae" containerName="revoke-keys" Sep 15 12:52:28.449600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449546 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae" containerName="revoke-keys" Sep 15 12:52:28.449600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449572 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55c1e87-a5e9-4609-a382-38af66d46b7a" containerName="payload-pre-processing" Sep 15 12:52:28.449600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449578 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55c1e87-a5e9-4609-a382-38af66d46b7a" containerName="payload-pre-processing" Sep 15 12:52:28.449600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449585 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" Sep 15 12:52:28.449600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449591 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" Sep 15 12:52:28.449600 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449598 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449603 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449609 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e056ffc-0e74-4f74-9da7-f5bbb9330b69" containerName="payload-processing" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449614 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e056ffc-0e74-4f74-9da7-f5bbb9330b69" containerName="payload-processing" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449674 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55c1e87-a5e9-4609-a382-38af66d46b7a" containerName="payload-pre-processing" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449681 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449688 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6593e5a4-ca36-463f-acc3-f9e83a106a93" containerName="istio-proxy" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449696 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="37560bc8-3bdf-4ce9-8f13-dcad8f03e6ae" containerName="revoke-keys" Sep 15 12:52:28.449799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.449707 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e056ffc-0e74-4f74-9da7-f5bbb9330b69" containerName="payload-processing" Sep 15 12:52:28.454434 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.454419 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" Sep 15 12:52:28.456930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.456887 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:52:28.457092 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.456899 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:52:28.459163 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.459143 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws"] Sep 15 12:52:28.546908 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.546877 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617f639c-1189-44a3-87f7-abc725146c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws\" (UID: \"617f639c-1189-44a3-87f7-abc725146c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" Sep 15 12:52:28.648570 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.648526 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617f639c-1189-44a3-87f7-abc725146c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws\" (UID: \"617f639c-1189-44a3-87f7-abc725146c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" Sep 15 12:52:28.649185 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.649160 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617f639c-1189-44a3-87f7-abc725146c21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws\" (UID: \"617f639c-1189-44a3-87f7-abc725146c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" Sep 15 12:52:28.768022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:28.767987 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" Sep 15 12:52:29.040423 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:29.040198 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws"] Sep 15 12:52:29.042868 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:52:29.042844 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617f639c_1189_44a3_87f7_abc725146c21.slice/crio-f17dee2e8e8ba68478fb3ac4679c7181853544222fab2bbc07c828171996d905 WatchSource:0}: Error finding container f17dee2e8e8ba68478fb3ac4679c7181853544222fab2bbc07c828171996d905: Status 404 returned error can't find the container with id f17dee2e8e8ba68478fb3ac4679c7181853544222fab2bbc07c828171996d905 Sep 15 12:52:29.182206 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:29.182170 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" event={"ID":"617f639c-1189-44a3-87f7-abc725146c21","Type":"ContainerStarted","Data":"f17dee2e8e8ba68478fb3ac4679c7181853544222fab2bbc07c828171996d905"} Sep 15 12:52:30.190500 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:30.190406 2550 generic.go:356] "Generic (PLEG): container finished" podID="617f639c-1189-44a3-87f7-abc725146c21" containerID="777ed2022b98f72f373f4f6c902ca21b7e98e54a4aaf36947b1034d2e4239e81" exitCode=0 Sep 15 12:52:30.190896 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:30.190491 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" event={"ID":"617f639c-1189-44a3-87f7-abc725146c21","Type":"ContainerDied","Data":"777ed2022b98f72f373f4f6c902ca21b7e98e54a4aaf36947b1034d2e4239e81"} Sep 15 12:52:31.342074 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:31.342047 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" Sep 15 12:52:31.476263 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:31.476156 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617f639c-1189-44a3-87f7-abc725146c21-maas-api-service-ca\") pod \"617f639c-1189-44a3-87f7-abc725146c21\" (UID: \"617f639c-1189-44a3-87f7-abc725146c21\") " Sep 15 12:52:31.476632 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:31.476601 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617f639c-1189-44a3-87f7-abc725146c21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "617f639c-1189-44a3-87f7-abc725146c21" (UID: "617f639c-1189-44a3-87f7-abc725146c21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:31.577351 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:31.577317 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/617f639c-1189-44a3-87f7-abc725146c21-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:32.204826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:32.204795 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" Sep 15 12:52:32.205014 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:32.204832 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws" event={"ID":"617f639c-1189-44a3-87f7-abc725146c21","Type":"ContainerDied","Data":"f17dee2e8e8ba68478fb3ac4679c7181853544222fab2bbc07c828171996d905"} Sep 15 12:52:32.205014 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:32.204885 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f17dee2e8e8ba68478fb3ac4679c7181853544222fab2bbc07c828171996d905" Sep 15 12:52:39.158760 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:39.158722 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m"] Sep 15 12:52:39.161170 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:39.161144 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a1ba7b-29824605-2g58m"] Sep 15 12:52:39.340104 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:39.340066 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws"] Sep 15 12:52:39.342247 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:39.342222 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a1ba7b-klqws"] Sep 15 12:52:41.154753 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:41.154717 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="617f639c-1189-44a3-87f7-abc725146c21" path="/var/lib/kubelet/pods/617f639c-1189-44a3-87f7-abc725146c21/volumes" Sep 15 12:52:41.155243 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:41.155149 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3675cf-8f8a-4a23-a956-6fd1ea98211f" path="/var/lib/kubelet/pods/6c3675cf-8f8a-4a23-a956-6fd1ea98211f/volumes" Sep 15 12:52:45.839114 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:45.839076 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh"] Sep 15 12:52:45.839581 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:45.839334 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" containerID="cri-o://e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651" gracePeriod=30 Sep 15 12:52:46.195648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.195566 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc"] Sep 15 12:52:46.196032 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.196020 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="617f639c-1189-44a3-87f7-abc725146c21" containerName="revoke-keys" Sep 15 12:52:46.196083 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.196035 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="617f639c-1189-44a3-87f7-abc725146c21" containerName="revoke-keys" Sep 15 12:52:46.196123 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.196093 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="617f639c-1189-44a3-87f7-abc725146c21" containerName="revoke-keys" Sep 15 12:52:46.200347 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.200332 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" Sep 15 12:52:46.202652 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.202629 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:52:46.202753 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.202629 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:52:46.208871 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.208796 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc"] Sep 15 12:52:46.320648 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.320613 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31127b5-01e5-4e22-9e3c-8348f3254482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc\" (UID: \"a31127b5-01e5-4e22-9e3c-8348f3254482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" Sep 15 12:52:46.422043 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.422007 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31127b5-01e5-4e22-9e3c-8348f3254482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc\" (UID: \"a31127b5-01e5-4e22-9e3c-8348f3254482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" Sep 15 12:52:46.422644 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.422625 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31127b5-01e5-4e22-9e3c-8348f3254482-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc\" (UID: \"a31127b5-01e5-4e22-9e3c-8348f3254482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" Sep 15 12:52:46.512823 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.512787 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" Sep 15 12:52:46.790455 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:46.790421 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc"] Sep 15 12:52:46.792394 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:52:46.792348 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31127b5_01e5_4e22_9e3c_8348f3254482.slice/crio-cef5c7fdcbfac7c0f3d3ccccd2c604765aad0c962d7191202886b811d5683bce WatchSource:0}: Error finding container cef5c7fdcbfac7c0f3d3ccccd2c604765aad0c962d7191202886b811d5683bce: Status 404 returned error can't find the container with id cef5c7fdcbfac7c0f3d3ccccd2c604765aad0c962d7191202886b811d5683bce Sep 15 12:52:47.295773 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:47.295738 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" event={"ID":"a31127b5-01e5-4e22-9e3c-8348f3254482","Type":"ContainerStarted","Data":"561178eaeb57ad3efa1b8a88aae94e9523f228a3b5273de204e8dd7ca8e115d6"} Sep 15 12:52:47.296147 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:47.295796 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" event={"ID":"a31127b5-01e5-4e22-9e3c-8348f3254482","Type":"ContainerStarted","Data":"cef5c7fdcbfac7c0f3d3ccccd2c604765aad0c962d7191202886b811d5683bce"} Sep 15 12:52:47.310861 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:47.310809 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" podStartSLOduration=1.3107922859999999 podStartE2EDuration="1.310792286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:52:47.309389862 +0000 UTC m=+1648.664236178" watchObservedRunningTime="2026-09-15 12:52:47.310792286 +0000 UTC m=+1648.665638608" Sep 15 12:52:48.303089 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:48.303052 2550 generic.go:356] "Generic (PLEG): container finished" podID="a31127b5-01e5-4e22-9e3c-8348f3254482" containerID="561178eaeb57ad3efa1b8a88aae94e9523f228a3b5273de204e8dd7ca8e115d6" exitCode=0 Sep 15 12:52:48.303528 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:48.303132 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" event={"ID":"a31127b5-01e5-4e22-9e3c-8348f3254482","Type":"ContainerDied","Data":"561178eaeb57ad3efa1b8a88aae94e9523f228a3b5273de204e8dd7ca8e115d6"} Sep 15 12:52:48.734626 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:48.734547 2550 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 12:52:48.734626 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:48.734602 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 12:52:49.449404 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:49.449362 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" Sep 15 12:52:49.556798 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:49.556762 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31127b5-01e5-4e22-9e3c-8348f3254482-maas-api-service-ca\") pod \"a31127b5-01e5-4e22-9e3c-8348f3254482\" (UID: \"a31127b5-01e5-4e22-9e3c-8348f3254482\") " Sep 15 12:52:49.557112 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:49.557089 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31127b5-01e5-4e22-9e3c-8348f3254482-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a31127b5-01e5-4e22-9e3c-8348f3254482" (UID: "a31127b5-01e5-4e22-9e3c-8348f3254482"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:49.658407 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:49.658294 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31127b5-01e5-4e22-9e3c-8348f3254482-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:50.317108 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:50.317072 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" Sep 15 12:52:50.317285 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:50.317110 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc" event={"ID":"a31127b5-01e5-4e22-9e3c-8348f3254482","Type":"ContainerDied","Data":"cef5c7fdcbfac7c0f3d3ccccd2c604765aad0c962d7191202886b811d5683bce"} Sep 15 12:52:50.317285 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:50.317160 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef5c7fdcbfac7c0f3d3ccccd2c604765aad0c962d7191202886b811d5683bce" Sep 15 12:52:51.102162 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.102096 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:52:51.174522 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174488 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-token\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.174522 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174528 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-socket\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.174766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174579 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-data\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.174766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174600 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-credential-socket\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.174766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174662 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5669a06-72ff-4194-9610-a5852ba6c6f8-istiod-ca-cert\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.174766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174681 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-envoy\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.174766 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174706 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-certs\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.175026 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174783 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr222\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-kube-api-access-lr222\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.175026 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174806 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-podinfo\") pod \"e5669a06-72ff-4194-9610-a5852ba6c6f8\" (UID: \"e5669a06-72ff-4194-9610-a5852ba6c6f8\") " Sep 15 12:52:51.175026 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174972 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:51.175026 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.174983 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:51.175242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.175011 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:51.175242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.175079 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:51.175242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.175186 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.175242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.175198 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.175242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.175208 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.175242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.175221 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.175242 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.175216 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5669a06-72ff-4194-9610-a5852ba6c6f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:51.176749 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.176720 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:51.177058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.177033 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:52:51.177439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.177416 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-kube-api-access-lr222" (OuterVolumeSpecName: "kube-api-access-lr222") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "kube-api-access-lr222". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:51.177495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.177416 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5669a06-72ff-4194-9610-a5852ba6c6f8" (UID: "e5669a06-72ff-4194-9610-a5852ba6c6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 12:52:51.275738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.275700 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5669a06-72ff-4194-9610-a5852ba6c6f8-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.275738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.275729 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.275738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.275738 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr222\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-kube-api-access-lr222\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.275738 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.275750 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.276067 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.275759 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5669a06-72ff-4194-9610-a5852ba6c6f8-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:51.325624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.325587 2550 generic.go:356] "Generic (PLEG): container finished" podID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerID="e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651" exitCode=0 Sep 15 12:52:51.325802 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.325635 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" Sep 15 12:52:51.325802 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.325667 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" event={"ID":"e5669a06-72ff-4194-9610-a5852ba6c6f8","Type":"ContainerDied","Data":"e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651"} Sep 15 12:52:51.325802 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.325721 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh" event={"ID":"e5669a06-72ff-4194-9610-a5852ba6c6f8","Type":"ContainerDied","Data":"eb090540250039ec29637315a1d31f1c57d2b0c8a8db699ade64b8f40e7f931d"} Sep 15 12:52:51.325802 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.325750 2550 scope.go:117] "RemoveContainer" containerID="e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651" Sep 15 12:52:51.336068 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.336042 2550 scope.go:117] "RemoveContainer" containerID="0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14" Sep 15 12:52:51.345679 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.345657 2550 scope.go:117] "RemoveContainer" containerID="e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651" Sep 15 12:52:51.346040 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:51.346024 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651\": container with ID starting with e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651 not found: ID does not exist" containerID="e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651" Sep 15 12:52:51.346080 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.346050 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651"} err="failed to get container status \"e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651\": rpc error: code = NotFound desc = could not find container \"e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651\": container with ID starting with e3bfe2b855c82808f6dce2123fff733912919332a025d3bd8a1697b263e89651 not found: ID does not exist" Sep 15 12:52:51.346080 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.346068 2550 scope.go:117] "RemoveContainer" containerID="0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14" Sep 15 12:52:51.346328 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:51.346307 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14\": container with ID starting with 0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14 not found: ID does not exist" containerID="0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14" Sep 15 12:52:51.346400 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.346332 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14"} err="failed to get container status \"0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14\": rpc error: code = NotFound desc = could not find container \"0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14\": container with ID starting with 0ee452bb9e948602140ffca050b995dba57e915ff217d844310b24825da44f14 not found: ID does not exist" Sep 15 12:52:51.347443 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.347421 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh"] Sep 15 12:52:51.351531 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:51.351503 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a1ba7b-openshift-default-8565468798-7kdnh"] Sep 15 12:52:53.154511 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:53.154471 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" path="/var/lib/kubelet/pods/e5669a06-72ff-4194-9610-a5852ba6c6f8/volumes" Sep 15 12:52:56.803017 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.802984 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7"] Sep 15 12:52:56.805532 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.805508 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d9e7a1-29824605-wxnw7"] Sep 15 12:52:56.835968 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.835939 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l"] Sep 15 12:52:56.836211 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.836185 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" podUID="0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" containerName="payload-processing" containerID="cri-o://64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6" gracePeriod=30 Sep 15 12:52:56.841932 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.841661 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6"] Sep 15 12:52:56.842071 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.841913 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" podUID="d3f068d9-8901-4b0a-aff1-ae6e45553cc3" containerName="payload-pre-processing" containerID="cri-o://08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5" gracePeriod=30 Sep 15 12:52:56.992631 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.992563 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc"] Sep 15 12:52:56.995902 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:56.995845 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d9e7a1-p5mwc"] Sep 15 12:52:57.119667 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.119640 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:52:57.144028 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.144001 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:52:57.155335 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.155306 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a31127b5-01e5-4e22-9e3c-8348f3254482" path="/var/lib/kubelet/pods/a31127b5-01e5-4e22-9e3c-8348f3254482/volumes" Sep 15 12:52:57.155616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.155601 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3945a01-d784-4175-97a5-de579badbba3" path="/var/lib/kubelet/pods/d3945a01-d784-4175-97a5-de579badbba3/volumes" Sep 15 12:52:57.239083 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.239052 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7btkt\" (UniqueName: \"kubernetes.io/projected/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-kube-api-access-7btkt\") pod \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " Sep 15 12:52:57.239256 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.239095 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqznh\" (UniqueName: \"kubernetes.io/projected/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-kube-api-access-gqznh\") pod \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " Sep 15 12:52:57.239256 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.239178 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-plugins-config-volume\") pod \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\" (UID: \"d3f068d9-8901-4b0a-aff1-ae6e45553cc3\") " Sep 15 12:52:57.239256 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.239211 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-plugins-config-volume\") pod \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\" (UID: \"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a\") " Sep 15 12:52:57.239665 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.239638 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3f068d9-8901-4b0a-aff1-ae6e45553cc3" (UID: "d3f068d9-8901-4b0a-aff1-ae6e45553cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:57.239783 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.239743 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" (UID: "0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:52:57.241253 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.241229 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-kube-api-access-7btkt" (OuterVolumeSpecName: "kube-api-access-7btkt") pod "d3f068d9-8901-4b0a-aff1-ae6e45553cc3" (UID: "d3f068d9-8901-4b0a-aff1-ae6e45553cc3"). InnerVolumeSpecName "kube-api-access-7btkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:57.241336 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.241291 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-kube-api-access-gqznh" (OuterVolumeSpecName: "kube-api-access-gqznh") pod "0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" (UID: "0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a"). InnerVolumeSpecName "kube-api-access-gqznh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:52:57.340744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.340650 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:57.340744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.340683 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:57.340744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.340697 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7btkt\" (UniqueName: \"kubernetes.io/projected/d3f068d9-8901-4b0a-aff1-ae6e45553cc3-kube-api-access-7btkt\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:57.340744 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.340711 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqznh\" (UniqueName: \"kubernetes.io/projected/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a-kube-api-access-gqznh\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:52:57.364174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.364141 2550 generic.go:356] "Generic (PLEG): container finished" podID="0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" containerID="64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6" exitCode=0 Sep 15 12:52:57.364344 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.364189 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" Sep 15 12:52:57.364344 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.364223 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" event={"ID":"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a","Type":"ContainerDied","Data":"64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6"} Sep 15 12:52:57.364344 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.364278 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l" event={"ID":"0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a","Type":"ContainerDied","Data":"a46bc022dcb83f7277ebd41d5db0d2c4019d873d44e982554a7c619814555cf2"} Sep 15 12:52:57.364344 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.364309 2550 scope.go:117] "RemoveContainer" containerID="64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6" Sep 15 12:52:57.365710 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.365674 2550 generic.go:356] "Generic (PLEG): container finished" podID="d3f068d9-8901-4b0a-aff1-ae6e45553cc3" containerID="08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5" exitCode=0 Sep 15 12:52:57.365836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.365717 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" Sep 15 12:52:57.365836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.365765 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" event={"ID":"d3f068d9-8901-4b0a-aff1-ae6e45553cc3","Type":"ContainerDied","Data":"08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5"} Sep 15 12:52:57.365836 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.365818 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6" event={"ID":"d3f068d9-8901-4b0a-aff1-ae6e45553cc3","Type":"ContainerDied","Data":"93ea3a90014eb2497f97e0dda9cd985131bf742bde32a82bbb769ec1b244a3d0"} Sep 15 12:52:57.374569 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.374546 2550 scope.go:117] "RemoveContainer" containerID="64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6" Sep 15 12:52:57.374867 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:57.374841 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6\": container with ID starting with 64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6 not found: ID does not exist" containerID="64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6" Sep 15 12:52:57.374935 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.374879 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6"} err="failed to get container status \"64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6\": rpc error: code = NotFound desc = could not find container \"64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6\": container with ID starting with 64ca13bbd8dbd414359be7cb1425ddccae1ec3ffa60a6e9f49dcf123e2276df6 not found: ID does not exist" Sep 15 12:52:57.374935 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.374902 2550 scope.go:117] "RemoveContainer" containerID="08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5" Sep 15 12:52:57.383734 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.383712 2550 scope.go:117] "RemoveContainer" containerID="08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5" Sep 15 12:52:57.384045 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:52:57.384021 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5\": container with ID starting with 08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5 not found: ID does not exist" containerID="08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5" Sep 15 12:52:57.384145 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.384053 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5"} err="failed to get container status \"08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5\": rpc error: code = NotFound desc = could not find container \"08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5\": container with ID starting with 08b864281b99677bd46c9363c5bc528f098dd8eb11f856dd8412e45bd6530eb5 not found: ID does not exist" Sep 15 12:52:57.386694 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.386674 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l"] Sep 15 12:52:57.388900 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.388881 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d9e7a1-58949bc577-btw8l"] Sep 15 12:52:57.397258 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.397236 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6"] Sep 15 12:52:57.403037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:57.403018 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d9e7a1-7c6498bd6b-2fzt6"] Sep 15 12:52:59.154220 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:59.154185 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" path="/var/lib/kubelet/pods/0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a/volumes" Sep 15 12:52:59.154634 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:52:59.154615 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3f068d9-8901-4b0a-aff1-ae6e45553cc3" path="/var/lib/kubelet/pods/d3f068d9-8901-4b0a-aff1-ae6e45553cc3/volumes" Sep 15 12:53:19.518361 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:19.518322 2550 scope.go:117] "RemoveContainer" containerID="40568d446b6dcaba15fcdc95ab6e73c8c30766f91f130828bd3c0b6fe66801ec" Sep 15 12:53:19.527094 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:19.527075 2550 scope.go:117] "RemoveContainer" containerID="e527454ff2662bf4b9cfa7c2a5190f5434ba4ce9456a371de6ae4e3ea0b4013a" Sep 15 12:53:19.535265 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:19.535243 2550 scope.go:117] "RemoveContainer" containerID="cb88277fc378bf61c8a5bab210dc50268519e7a1e8b281117eba47e73dd2f1b5" Sep 15 12:53:31.951827 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.951784 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x"] Sep 15 12:53:31.952298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952238 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" containerName="payload-processing" Sep 15 12:53:31.952298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952253 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" containerName="payload-processing" Sep 15 12:53:31.952298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952265 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" Sep 15 12:53:31.952298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952271 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" Sep 15 12:53:31.952298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952279 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f068d9-8901-4b0a-aff1-ae6e45553cc3" containerName="payload-pre-processing" Sep 15 12:53:31.952298 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952284 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f068d9-8901-4b0a-aff1-ae6e45553cc3" containerName="payload-pre-processing" Sep 15 12:53:31.952517 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952305 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a31127b5-01e5-4e22-9e3c-8348f3254482" containerName="revoke-keys" Sep 15 12:53:31.952517 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952310 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a31127b5-01e5-4e22-9e3c-8348f3254482" containerName="revoke-keys" Sep 15 12:53:31.952517 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952420 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" Sep 15 12:53:31.952517 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952429 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f068d9-8901-4b0a-aff1-ae6e45553cc3" containerName="payload-pre-processing" Sep 15 12:53:31.952517 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952438 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a31127b5-01e5-4e22-9e3c-8348f3254482" containerName="revoke-keys" Sep 15 12:53:31.952517 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.952446 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb367ea-98ac-4d2e-9d20-fdd0b0e9148a" containerName="payload-processing" Sep 15 12:53:31.955588 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.955573 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" Sep 15 12:53:31.957889 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.957867 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:53:31.958182 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.957866 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:53:31.962184 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:31.962161 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x"] Sep 15 12:53:32.078440 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:32.078362 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59062788-ed65-4d54-b224-efc18faf5c3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x\" (UID: \"59062788-ed65-4d54-b224-efc18faf5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" Sep 15 12:53:32.179846 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:32.179805 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59062788-ed65-4d54-b224-efc18faf5c3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x\" (UID: \"59062788-ed65-4d54-b224-efc18faf5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" Sep 15 12:53:32.180474 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:32.180452 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59062788-ed65-4d54-b224-efc18faf5c3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x\" (UID: \"59062788-ed65-4d54-b224-efc18faf5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" Sep 15 12:53:32.268055 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:32.268013 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" Sep 15 12:53:32.540078 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:32.540049 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x"] Sep 15 12:53:32.542328 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:53:32.542277 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59062788_ed65_4d54_b224_efc18faf5c3f.slice/crio-53f8da37baeea11b18b210d3febf5f51ef20c781c8869d9f23d796e2be6acc7c WatchSource:0}: Error finding container 53f8da37baeea11b18b210d3febf5f51ef20c781c8869d9f23d796e2be6acc7c: Status 404 returned error can't find the container with id 53f8da37baeea11b18b210d3febf5f51ef20c781c8869d9f23d796e2be6acc7c Sep 15 12:53:32.577116 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:32.577082 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" event={"ID":"59062788-ed65-4d54-b224-efc18faf5c3f","Type":"ContainerStarted","Data":"53f8da37baeea11b18b210d3febf5f51ef20c781c8869d9f23d796e2be6acc7c"} Sep 15 12:53:33.584026 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:33.583941 2550 generic.go:356] "Generic (PLEG): container finished" podID="59062788-ed65-4d54-b224-efc18faf5c3f" containerID="a8b433bbe6d396ca1a283b1d3a781e8b508dbe9cfd251e6a2f951c5d93a6e455" exitCode=0 Sep 15 12:53:33.584415 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:33.584023 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" event={"ID":"59062788-ed65-4d54-b224-efc18faf5c3f","Type":"ContainerDied","Data":"a8b433bbe6d396ca1a283b1d3a781e8b508dbe9cfd251e6a2f951c5d93a6e455"} Sep 15 12:53:34.731075 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:34.731049 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" Sep 15 12:53:34.904997 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:34.904910 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59062788-ed65-4d54-b224-efc18faf5c3f-maas-api-service-ca\") pod \"59062788-ed65-4d54-b224-efc18faf5c3f\" (UID: \"59062788-ed65-4d54-b224-efc18faf5c3f\") " Sep 15 12:53:34.905286 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:34.905262 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59062788-ed65-4d54-b224-efc18faf5c3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59062788-ed65-4d54-b224-efc18faf5c3f" (UID: "59062788-ed65-4d54-b224-efc18faf5c3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:53:35.006676 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:35.006639 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59062788-ed65-4d54-b224-efc18faf5c3f-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:35.596613 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:35.596582 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" Sep 15 12:53:35.596613 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:35.596604 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x" event={"ID":"59062788-ed65-4d54-b224-efc18faf5c3f","Type":"ContainerDied","Data":"53f8da37baeea11b18b210d3febf5f51ef20c781c8869d9f23d796e2be6acc7c"} Sep 15 12:53:35.596826 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:35.596645 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53f8da37baeea11b18b210d3febf5f51ef20c781c8869d9f23d796e2be6acc7c" Sep 15 12:53:47.467119 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.467019 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb"] Sep 15 12:53:47.469058 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.469035 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-11f865-29824605-xzfsb"] Sep 15 12:53:47.496420 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.496371 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw"] Sep 15 12:53:47.496686 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.496632 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" podUID="34dfb603-5cee-478b-9fe1-cec070ff2e18" containerName="payload-processing" containerID="cri-o://1891d26b0b817444c8ee46e8aa5332487fa29ecb2d2fea0364a4182aff4c89a3" gracePeriod=30 Sep 15 12:53:47.506964 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.506936 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz"] Sep 15 12:53:47.507306 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.507278 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" podUID="2a1777c2-a59b-497e-b179-a8f4785d56a8" containerName="payload-pre-processing" containerID="cri-o://fc78f9084f894c330efce8a61dfb11d510cd25c858652c7ebbe8b306049b84cf" gracePeriod=30 Sep 15 12:53:47.665457 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.665422 2550 generic.go:356] "Generic (PLEG): container finished" podID="2a1777c2-a59b-497e-b179-a8f4785d56a8" containerID="fc78f9084f894c330efce8a61dfb11d510cd25c858652c7ebbe8b306049b84cf" exitCode=0 Sep 15 12:53:47.665590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.665499 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" event={"ID":"2a1777c2-a59b-497e-b179-a8f4785d56a8","Type":"ContainerDied","Data":"fc78f9084f894c330efce8a61dfb11d510cd25c858652c7ebbe8b306049b84cf"} Sep 15 12:53:47.667147 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.667125 2550 generic.go:356] "Generic (PLEG): container finished" podID="34dfb603-5cee-478b-9fe1-cec070ff2e18" containerID="1891d26b0b817444c8ee46e8aa5332487fa29ecb2d2fea0364a4182aff4c89a3" exitCode=0 Sep 15 12:53:47.667347 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.667189 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" event={"ID":"34dfb603-5cee-478b-9fe1-cec070ff2e18","Type":"ContainerDied","Data":"1891d26b0b817444c8ee46e8aa5332487fa29ecb2d2fea0364a4182aff4c89a3"} Sep 15 12:53:47.674052 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.674027 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x"] Sep 15 12:53:47.679064 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.679041 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-11f865-lch9x"] Sep 15 12:53:47.785174 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.785149 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:53:47.811530 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.811494 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:53:47.827945 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.827909 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5mtj\" (UniqueName: \"kubernetes.io/projected/34dfb603-5cee-478b-9fe1-cec070ff2e18-kube-api-access-m5mtj\") pod \"34dfb603-5cee-478b-9fe1-cec070ff2e18\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " Sep 15 12:53:47.828265 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.828245 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99j4r\" (UniqueName: \"kubernetes.io/projected/2a1777c2-a59b-497e-b179-a8f4785d56a8-kube-api-access-99j4r\") pod \"2a1777c2-a59b-497e-b179-a8f4785d56a8\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " Sep 15 12:53:47.828466 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.828448 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34dfb603-5cee-478b-9fe1-cec070ff2e18-plugins-config-volume\") pod \"34dfb603-5cee-478b-9fe1-cec070ff2e18\" (UID: \"34dfb603-5cee-478b-9fe1-cec070ff2e18\") " Sep 15 12:53:47.828609 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.828593 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1777c2-a59b-497e-b179-a8f4785d56a8-plugins-config-volume\") pod \"2a1777c2-a59b-497e-b179-a8f4785d56a8\" (UID: \"2a1777c2-a59b-497e-b179-a8f4785d56a8\") " Sep 15 12:53:47.829548 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.829520 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1777c2-a59b-497e-b179-a8f4785d56a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a1777c2-a59b-497e-b179-a8f4785d56a8" (UID: "2a1777c2-a59b-497e-b179-a8f4785d56a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:53:47.830338 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.830314 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34dfb603-5cee-478b-9fe1-cec070ff2e18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34dfb603-5cee-478b-9fe1-cec070ff2e18" (UID: "34dfb603-5cee-478b-9fe1-cec070ff2e18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:53:47.834874 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.834840 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1777c2-a59b-497e-b179-a8f4785d56a8-kube-api-access-99j4r" (OuterVolumeSpecName: "kube-api-access-99j4r") pod "2a1777c2-a59b-497e-b179-a8f4785d56a8" (UID: "2a1777c2-a59b-497e-b179-a8f4785d56a8"). InnerVolumeSpecName "kube-api-access-99j4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:53:47.836852 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.836826 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34dfb603-5cee-478b-9fe1-cec070ff2e18-kube-api-access-m5mtj" (OuterVolumeSpecName: "kube-api-access-m5mtj") pod "34dfb603-5cee-478b-9fe1-cec070ff2e18" (UID: "34dfb603-5cee-478b-9fe1-cec070ff2e18"). InnerVolumeSpecName "kube-api-access-m5mtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:53:47.930509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.930467 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99j4r\" (UniqueName: \"kubernetes.io/projected/2a1777c2-a59b-497e-b179-a8f4785d56a8-kube-api-access-99j4r\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:47.930509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.930502 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34dfb603-5cee-478b-9fe1-cec070ff2e18-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:47.930509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.930511 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1777c2-a59b-497e-b179-a8f4785d56a8-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:47.930509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:47.930521 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5mtj\" (UniqueName: \"kubernetes.io/projected/34dfb603-5cee-478b-9fe1-cec070ff2e18-kube-api-access-m5mtj\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:48.674533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.674498 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" Sep 15 12:53:48.674533 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.674518 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw" event={"ID":"34dfb603-5cee-478b-9fe1-cec070ff2e18","Type":"ContainerDied","Data":"b9fde5e0a10cb5cfa8ac2c71d7f2e13364f8d75c8476fbcd741306fe87e8bd26"} Sep 15 12:53:48.675051 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.674574 2550 scope.go:117] "RemoveContainer" containerID="1891d26b0b817444c8ee46e8aa5332487fa29ecb2d2fea0364a4182aff4c89a3" Sep 15 12:53:48.676189 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.676162 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" event={"ID":"2a1777c2-a59b-497e-b179-a8f4785d56a8","Type":"ContainerDied","Data":"43be24fddf2039e6bee818c6fb2d6b1d564bc802e71c49ea809ec4808467222c"} Sep 15 12:53:48.676393 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.676186 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz" Sep 15 12:53:48.687443 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.687425 2550 scope.go:117] "RemoveContainer" containerID="fc78f9084f894c330efce8a61dfb11d510cd25c858652c7ebbe8b306049b84cf" Sep 15 12:53:48.701748 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.701724 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz"] Sep 15 12:53:48.705124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.705104 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-11f865-8699c4684b-nh6gz"] Sep 15 12:53:48.715683 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.715653 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw"] Sep 15 12:53:48.717561 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:48.717540 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-11f865-d44cf7d69-mwpcw"] Sep 15 12:53:49.154141 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:49.154106 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24db67b2-af79-4efd-8ec5-11cf996665a9" path="/var/lib/kubelet/pods/24db67b2-af79-4efd-8ec5-11cf996665a9/volumes" Sep 15 12:53:49.154481 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:49.154466 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1777c2-a59b-497e-b179-a8f4785d56a8" path="/var/lib/kubelet/pods/2a1777c2-a59b-497e-b179-a8f4785d56a8/volumes" Sep 15 12:53:49.154830 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:49.154816 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34dfb603-5cee-478b-9fe1-cec070ff2e18" path="/var/lib/kubelet/pods/34dfb603-5cee-478b-9fe1-cec070ff2e18/volumes" Sep 15 12:53:49.155105 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:49.155094 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59062788-ed65-4d54-b224-efc18faf5c3f" path="/var/lib/kubelet/pods/59062788-ed65-4d54-b224-efc18faf5c3f/volumes" Sep 15 12:53:53.872818 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:53.872783 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8"] Sep 15 12:53:53.873204 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:53.873042 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" containerID="cri-o://5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11" gracePeriod=30 Sep 15 12:53:54.387450 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:54.387410 2550 patch_prober.go:28] interesting pod/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 12:53:54.387624 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:54.387486 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 12:53:59.129323 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.129295 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:53:59.241693 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241659 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-socket\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.241860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241730 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4jz6\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-kube-api-access-x4jz6\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.241860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241758 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-podinfo\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.241860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241783 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-data\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.241860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241805 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-envoy\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.241860 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241829 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-token\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.242128 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241868 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-credential-socket\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.242128 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241891 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-certs\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.242128 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.241977 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istiod-ca-cert\") pod \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\" (UID: \"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3\") " Sep 15 12:53:59.242293 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242117 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:53:59.242293 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242161 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:53:59.242447 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242430 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:53:59.242514 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242490 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:53:59.242883 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242807 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.242883 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242837 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.242883 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242851 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.243060 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.242972 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:53:59.244123 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.244097 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:53:59.244469 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.244448 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 12:53:59.244574 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.244552 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:53:59.244797 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.244781 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-kube-api-access-x4jz6" (OuterVolumeSpecName: "kube-api-access-x4jz6") pod "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" (UID: "59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3"). InnerVolumeSpecName "kube-api-access-x4jz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:53:59.344279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.344241 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.344279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.344277 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.344279 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.344287 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jz6\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-kube-api-access-x4jz6\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.344550 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.344297 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.344550 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.344306 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.344550 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.344314 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:53:59.749037 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.749004 2550 generic.go:356] "Generic (PLEG): container finished" podID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerID="5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11" exitCode=0 Sep 15 12:53:59.749195 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.749072 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" Sep 15 12:53:59.749195 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.749084 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" event={"ID":"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3","Type":"ContainerDied","Data":"5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11"} Sep 15 12:53:59.749195 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.749146 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8" event={"ID":"59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3","Type":"ContainerDied","Data":"5acdc65ae667606b9fc643c922a8fc3165534847aaffd6d3ecb3bf871e595c80"} Sep 15 12:53:59.749195 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.749177 2550 scope.go:117] "RemoveContainer" containerID="5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11" Sep 15 12:53:59.758896 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.758878 2550 scope.go:117] "RemoveContainer" containerID="3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c" Sep 15 12:53:59.766822 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.766803 2550 scope.go:117] "RemoveContainer" containerID="5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11" Sep 15 12:53:59.767086 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:53:59.767065 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11\": container with ID starting with 5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11 not found: ID does not exist" containerID="5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11" Sep 15 12:53:59.767151 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.767094 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11"} err="failed to get container status \"5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11\": rpc error: code = NotFound desc = could not find container \"5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11\": container with ID starting with 5ff16d03614a79f5f55b6d82c8b5ee776eeccbfff52610720d21c1900e293c11 not found: ID does not exist" Sep 15 12:53:59.767151 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.767111 2550 scope.go:117] "RemoveContainer" containerID="3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c" Sep 15 12:53:59.767354 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:53:59.767335 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c\": container with ID starting with 3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c not found: ID does not exist" containerID="3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c" Sep 15 12:53:59.767422 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.767357 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c"} err="failed to get container status \"3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c\": rpc error: code = NotFound desc = could not find container \"3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c\": container with ID starting with 3c9afa88122942353d121aee9d4b8daa7382209a8e7782a114f14c1fe36cef4c not found: ID does not exist" Sep 15 12:53:59.771100 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.771076 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8"] Sep 15 12:53:59.774710 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:53:59.774690 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-11f865-openshift-default-547c87b86-jbnl8"] Sep 15 12:54:01.153837 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:01.153802 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" path="/var/lib/kubelet/pods/59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3/volumes" Sep 15 12:54:02.975543 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.975513 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k"] Sep 15 12:54:02.975996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.975939 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1777c2-a59b-497e-b179-a8f4785d56a8" containerName="payload-pre-processing" Sep 15 12:54:02.975996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.975953 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1777c2-a59b-497e-b179-a8f4785d56a8" containerName="payload-pre-processing" Sep 15 12:54:02.975996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.975964 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34dfb603-5cee-478b-9fe1-cec070ff2e18" containerName="payload-processing" Sep 15 12:54:02.975996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.975971 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="34dfb603-5cee-478b-9fe1-cec070ff2e18" containerName="payload-processing" Sep 15 12:54:02.975996 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.975995 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976000 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976007 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59062788-ed65-4d54-b224-efc18faf5c3f" containerName="revoke-keys" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976013 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59062788-ed65-4d54-b224-efc18faf5c3f" containerName="revoke-keys" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976024 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976033 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976040 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976045 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976106 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="34dfb603-5cee-478b-9fe1-cec070ff2e18" containerName="payload-processing" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976117 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976123 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1777c2-a59b-497e-b179-a8f4785d56a8" containerName="payload-pre-processing" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976130 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59062788-ed65-4d54-b224-efc18faf5c3f" containerName="revoke-keys" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976136 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5669a06-72ff-4194-9610-a5852ba6c6f8" containerName="istio-proxy" Sep 15 12:54:02.976165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.976143 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a5d410-69b2-4ae2-8be3-a9b6e5a1c9f3" containerName="istio-proxy" Sep 15 12:54:02.980144 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.980127 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" Sep 15 12:54:02.982439 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.982417 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:54:02.982555 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.982419 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:54:02.986656 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:02.986631 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k"] Sep 15 12:54:03.079544 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.079509 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aebb825-a624-4e8f-a3a0-049ae586cf6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k\" (UID: \"0aebb825-a624-4e8f-a3a0-049ae586cf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" Sep 15 12:54:03.180510 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.180477 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aebb825-a624-4e8f-a3a0-049ae586cf6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k\" (UID: \"0aebb825-a624-4e8f-a3a0-049ae586cf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" Sep 15 12:54:03.181085 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.181066 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aebb825-a624-4e8f-a3a0-049ae586cf6e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k\" (UID: \"0aebb825-a624-4e8f-a3a0-049ae586cf6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" Sep 15 12:54:03.291819 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.291767 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" Sep 15 12:54:03.561594 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.561569 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k"] Sep 15 12:54:03.563111 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:54:03.563081 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aebb825_a624_4e8f_a3a0_049ae586cf6e.slice/crio-9cb722eccbd3b80c17780df9cf552d9798ffa05767866f62a043bc1dd44b52da WatchSource:0}: Error finding container 9cb722eccbd3b80c17780df9cf552d9798ffa05767866f62a043bc1dd44b52da: Status 404 returned error can't find the container with id 9cb722eccbd3b80c17780df9cf552d9798ffa05767866f62a043bc1dd44b52da Sep 15 12:54:03.773637 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.773604 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" event={"ID":"0aebb825-a624-4e8f-a3a0-049ae586cf6e","Type":"ContainerStarted","Data":"35368dd508fe533699c1225882c1b9388f4161f618ad929756a230a55446c4c8"} Sep 15 12:54:03.773771 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.773658 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" event={"ID":"0aebb825-a624-4e8f-a3a0-049ae586cf6e","Type":"ContainerStarted","Data":"9cb722eccbd3b80c17780df9cf552d9798ffa05767866f62a043bc1dd44b52da"} Sep 15 12:54:03.789061 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:03.789008 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" podStartSLOduration=1.7889929850000001 podStartE2EDuration="1.788992985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:54:03.787141652 +0000 UTC m=+1725.141987977" watchObservedRunningTime="2026-09-15 12:54:03.788992985 +0000 UTC m=+1725.143839354" Sep 15 12:54:04.780917 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:04.780880 2550 generic.go:356] "Generic (PLEG): container finished" podID="0aebb825-a624-4e8f-a3a0-049ae586cf6e" containerID="35368dd508fe533699c1225882c1b9388f4161f618ad929756a230a55446c4c8" exitCode=0 Sep 15 12:54:04.781355 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:04.780939 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" event={"ID":"0aebb825-a624-4e8f-a3a0-049ae586cf6e","Type":"ContainerDied","Data":"35368dd508fe533699c1225882c1b9388f4161f618ad929756a230a55446c4c8"} Sep 15 12:54:05.922977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:05.922952 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" Sep 15 12:54:06.002495 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:06.002461 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aebb825-a624-4e8f-a3a0-049ae586cf6e-maas-api-service-ca\") pod \"0aebb825-a624-4e8f-a3a0-049ae586cf6e\" (UID: \"0aebb825-a624-4e8f-a3a0-049ae586cf6e\") " Sep 15 12:54:06.002858 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:06.002833 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aebb825-a624-4e8f-a3a0-049ae586cf6e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0aebb825-a624-4e8f-a3a0-049ae586cf6e" (UID: "0aebb825-a624-4e8f-a3a0-049ae586cf6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:54:06.104270 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:06.104179 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aebb825-a624-4e8f-a3a0-049ae586cf6e-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:54:06.793505 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:06.793477 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" Sep 15 12:54:06.793505 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:06.793488 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k" event={"ID":"0aebb825-a624-4e8f-a3a0-049ae586cf6e","Type":"ContainerDied","Data":"9cb722eccbd3b80c17780df9cf552d9798ffa05767866f62a043bc1dd44b52da"} Sep 15 12:54:06.793718 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:06.793533 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cb722eccbd3b80c17780df9cf552d9798ffa05767866f62a043bc1dd44b52da" Sep 15 12:54:18.581074 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.581040 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk"] Sep 15 12:54:18.581521 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.581265 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" podUID="72b2453a-58d8-4eaa-8f2e-9e2075dd695b" containerName="payload-processing" containerID="cri-o://f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2" gracePeriod=30 Sep 15 12:54:18.592820 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.592787 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh"] Sep 15 12:54:18.593116 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.593019 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" podUID="139bd5e4-11c1-4561-a678-0f1322d3c5a8" containerName="payload-pre-processing" containerID="cri-o://88ec35cdfffe2a0b6ef6dcb74be65dda8b2c0919488055d27e3421402dc39426" gracePeriod=30 Sep 15 12:54:18.763544 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.763512 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k"] Sep 15 12:54:18.768761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.768732 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ee96ec-tj48k"] Sep 15 12:54:18.852735 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.852702 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:54:18.870587 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.870557 2550 generic.go:356] "Generic (PLEG): container finished" podID="139bd5e4-11c1-4561-a678-0f1322d3c5a8" containerID="88ec35cdfffe2a0b6ef6dcb74be65dda8b2c0919488055d27e3421402dc39426" exitCode=0 Sep 15 12:54:18.871050 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.870633 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" event={"ID":"139bd5e4-11c1-4561-a678-0f1322d3c5a8","Type":"ContainerDied","Data":"88ec35cdfffe2a0b6ef6dcb74be65dda8b2c0919488055d27e3421402dc39426"} Sep 15 12:54:18.872598 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.872577 2550 generic.go:356] "Generic (PLEG): container finished" podID="72b2453a-58d8-4eaa-8f2e-9e2075dd695b" containerID="f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2" exitCode=0 Sep 15 12:54:18.872725 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.872613 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" event={"ID":"72b2453a-58d8-4eaa-8f2e-9e2075dd695b","Type":"ContainerDied","Data":"f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2"} Sep 15 12:54:18.872831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.872767 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" event={"ID":"72b2453a-58d8-4eaa-8f2e-9e2075dd695b","Type":"ContainerDied","Data":"90f46ecd1b0711c30524800df11844ffd612af3d3a504e1f953f30fe78726929"} Sep 15 12:54:18.872831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.872633 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk" Sep 15 12:54:18.872831 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.872797 2550 scope.go:117] "RemoveContainer" containerID="f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2" Sep 15 12:54:18.885137 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.884988 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:54:18.885137 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.885122 2550 scope.go:117] "RemoveContainer" containerID="f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2" Sep 15 12:54:18.885471 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:54:18.885447 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2\": container with ID starting with f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2 not found: ID does not exist" containerID="f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2" Sep 15 12:54:18.885546 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:18.885483 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2"} err="failed to get container status \"f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2\": rpc error: code = NotFound desc = could not find container \"f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2\": container with ID starting with f7b9bc6392f056e6ba3093c2fc808cb23c4fbc70a6307c0fe2d86d97b14dc8d2 not found: ID does not exist" Sep 15 12:54:19.027956 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.027925 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx8gv\" (UniqueName: \"kubernetes.io/projected/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-kube-api-access-lx8gv\") pod \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " Sep 15 12:54:19.028142 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.028075 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5b6l\" (UniqueName: \"kubernetes.io/projected/139bd5e4-11c1-4561-a678-0f1322d3c5a8-kube-api-access-v5b6l\") pod \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " Sep 15 12:54:19.028142 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.028121 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139bd5e4-11c1-4561-a678-0f1322d3c5a8-plugins-config-volume\") pod \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\" (UID: \"139bd5e4-11c1-4561-a678-0f1322d3c5a8\") " Sep 15 12:54:19.028283 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.028156 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-plugins-config-volume\") pod \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\" (UID: \"72b2453a-58d8-4eaa-8f2e-9e2075dd695b\") " Sep 15 12:54:19.028580 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.028554 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139bd5e4-11c1-4561-a678-0f1322d3c5a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "139bd5e4-11c1-4561-a678-0f1322d3c5a8" (UID: "139bd5e4-11c1-4561-a678-0f1322d3c5a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:54:19.028670 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.028585 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72b2453a-58d8-4eaa-8f2e-9e2075dd695b" (UID: "72b2453a-58d8-4eaa-8f2e-9e2075dd695b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:54:19.030022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.029996 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-kube-api-access-lx8gv" (OuterVolumeSpecName: "kube-api-access-lx8gv") pod "72b2453a-58d8-4eaa-8f2e-9e2075dd695b" (UID: "72b2453a-58d8-4eaa-8f2e-9e2075dd695b"). InnerVolumeSpecName "kube-api-access-lx8gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:54:19.030094 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.030062 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139bd5e4-11c1-4561-a678-0f1322d3c5a8-kube-api-access-v5b6l" (OuterVolumeSpecName: "kube-api-access-v5b6l") pod "139bd5e4-11c1-4561-a678-0f1322d3c5a8" (UID: "139bd5e4-11c1-4561-a678-0f1322d3c5a8"). InnerVolumeSpecName "kube-api-access-v5b6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:54:19.130021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.129917 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5b6l\" (UniqueName: \"kubernetes.io/projected/139bd5e4-11c1-4561-a678-0f1322d3c5a8-kube-api-access-v5b6l\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:54:19.130021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.129951 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/139bd5e4-11c1-4561-a678-0f1322d3c5a8-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:54:19.130021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.129963 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:54:19.130021 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.129974 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8gv\" (UniqueName: \"kubernetes.io/projected/72b2453a-58d8-4eaa-8f2e-9e2075dd695b-kube-api-access-lx8gv\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:54:19.154535 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.154500 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aebb825-a624-4e8f-a3a0-049ae586cf6e" path="/var/lib/kubelet/pods/0aebb825-a624-4e8f-a3a0-049ae586cf6e/volumes" Sep 15 12:54:19.189098 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.189061 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk"] Sep 15 12:54:19.191217 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.191197 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ee96ec-76578769b5-krzxk"] Sep 15 12:54:19.585329 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.585298 2550 scope.go:117] "RemoveContainer" containerID="88ec35cdfffe2a0b6ef6dcb74be65dda8b2c0919488055d27e3421402dc39426" Sep 15 12:54:19.594068 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.594046 2550 scope.go:117] "RemoveContainer" containerID="54c5fd3f8ab4bfb99079f22edef55c079706352d325641868dbbff367e98a99f" Sep 15 12:54:19.608822 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.608801 2550 scope.go:117] "RemoveContainer" containerID="0873735986d0dd54d40e5b612435ca83e13937e0097e07feb7f250b372ba52a3" Sep 15 12:54:19.879633 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.879340 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" Sep 15 12:54:19.879991 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.879351 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh" event={"ID":"139bd5e4-11c1-4561-a678-0f1322d3c5a8","Type":"ContainerDied","Data":"64fb797de48a93a3e47e124f0f6b32675d12b8edbbbcc746bed60e838126a7f7"} Sep 15 12:54:19.913323 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.913284 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh"] Sep 15 12:54:19.925461 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:19.919607 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ee96ec-5ddbc7dbf8-fbvhh"] Sep 15 12:54:21.154980 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:21.154934 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="139bd5e4-11c1-4561-a678-0f1322d3c5a8" path="/var/lib/kubelet/pods/139bd5e4-11c1-4561-a678-0f1322d3c5a8/volumes" Sep 15 12:54:21.155509 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:54:21.155487 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b2453a-58d8-4eaa-8f2e-9e2075dd695b" path="/var/lib/kubelet/pods/72b2453a-58d8-4eaa-8f2e-9e2075dd695b/volumes" Sep 15 12:55:19.250616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:55:19.250587 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:55:19.252767 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:55:19.252744 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 12:57:29.367213 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367170 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8"] Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367605 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b2453a-58d8-4eaa-8f2e-9e2075dd695b" containerName="payload-processing" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367619 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b2453a-58d8-4eaa-8f2e-9e2075dd695b" containerName="payload-processing" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367636 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aebb825-a624-4e8f-a3a0-049ae586cf6e" containerName="revoke-keys" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367642 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aebb825-a624-4e8f-a3a0-049ae586cf6e" containerName="revoke-keys" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367656 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="139bd5e4-11c1-4561-a678-0f1322d3c5a8" containerName="payload-pre-processing" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367661 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="139bd5e4-11c1-4561-a678-0f1322d3c5a8" containerName="payload-pre-processing" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367716 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="139bd5e4-11c1-4561-a678-0f1322d3c5a8" containerName="payload-pre-processing" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367724 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aebb825-a624-4e8f-a3a0-049ae586cf6e" containerName="revoke-keys" Sep 15 12:57:29.367794 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.367733 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b2453a-58d8-4eaa-8f2e-9e2075dd695b" containerName="payload-processing" Sep 15 12:57:29.371032 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.371011 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.373198 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.373170 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0171b-openshift-default-dockercfg-6x7dn\"" Sep 15 12:57:29.381696 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.381673 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8"] Sep 15 12:57:29.419847 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.419814 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxr6\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-kube-api-access-vtxr6\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.419862 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-socket\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.419917 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-envoy\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.419953 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-token\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.419993 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-certs\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420149 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.420016 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-data\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420149 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.420045 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istiod-ca-cert\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420149 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.420074 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-credential-socket\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.420149 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.420097 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-podinfo\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.520771 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520721 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-certs\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.520771 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520773 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-data\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520800 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istiod-ca-cert\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520830 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-credential-socket\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520859 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-podinfo\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520894 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxr6\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-kube-api-access-vtxr6\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520939 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-socket\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521022 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.520979 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-envoy\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.521030 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-token\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.521170 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-certs\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.521200 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-data\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521403 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.521271 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-socket\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521617 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.521527 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istiod-ca-cert\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.521617 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.521605 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-credential-socket\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.523260 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.523234 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-envoy\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.523412 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.523373 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-podinfo\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.529212 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.529186 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-token\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.529322 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.529307 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxr6\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-kube-api-access-vtxr6\") pod \"e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.682187 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.682091 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:29.959147 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.959121 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8"] Sep 15 12:57:29.961273 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:57:29.961242 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b316189_29eb_4bc7_95eb_d0ea6325c2a0.slice/crio-b13e74d72033497c882e151d35f5c1cd70580efa7956b4f8b05fe884c1901362 WatchSource:0}: Error finding container b13e74d72033497c882e151d35f5c1cd70580efa7956b4f8b05fe884c1901362: Status 404 returned error can't find the container with id b13e74d72033497c882e151d35f5c1cd70580efa7956b4f8b05fe884c1901362 Sep 15 12:57:29.963799 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.963751 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:29.963927 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.963840 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:29.963927 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:29.963893 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:30.898367 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:30.898329 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" event={"ID":"6b316189-29eb-4bc7-95eb-d0ea6325c2a0","Type":"ContainerStarted","Data":"083f04de6a189d2b595c28dfa3c141fbe87d9766c7c4c8607dcf3cbb04db31f6"} Sep 15 12:57:30.898773 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:30.898405 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" event={"ID":"6b316189-29eb-4bc7-95eb-d0ea6325c2a0","Type":"ContainerStarted","Data":"b13e74d72033497c882e151d35f5c1cd70580efa7956b4f8b05fe884c1901362"} Sep 15 12:57:30.918768 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:30.918711 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podStartSLOduration=1.9186938900000001 podStartE2EDuration="1.91869389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:57:30.917058634 +0000 UTC m=+1932.271904967" watchObservedRunningTime="2026-09-15 12:57:30.91869389 +0000 UTC m=+1932.273540213" Sep 15 12:57:31.682946 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:31.682904 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:31.688057 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:31.688030 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:31.903454 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:31.903419 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:31.904513 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:31.904491 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:35.138128 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.138089 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9"] Sep 15 12:57:35.141786 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.141770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.144126 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.144104 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0171b\"" Sep 15 12:57:35.144248 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.144146 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0171b-dockercfg-j67mc\"" Sep 15 12:57:35.157795 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.157768 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9"] Sep 15 12:57:35.286955 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.286919 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/a3f76539-5719-467b-b2be-47f030ca9160-kube-api-access-bhzp2\") pod \"payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.287161 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.286973 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f76539-5719-467b-b2be-47f030ca9160-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.388293 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.388198 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/a3f76539-5719-467b-b2be-47f030ca9160-kube-api-access-bhzp2\") pod \"payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.388293 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.388252 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f76539-5719-467b-b2be-47f030ca9160-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.388923 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.388901 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f76539-5719-467b-b2be-47f030ca9160-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.395721 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.395693 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/a3f76539-5719-467b-b2be-47f030ca9160-kube-api-access-bhzp2\") pod \"payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.454045 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.454012 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.724981 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.724957 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9"] Sep 15 12:57:35.727551 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:57:35.727517 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f76539_5719_467b_b2be_47f030ca9160.slice/crio-567f0b151e82574b6353852636e3cb2b7534941899a86e459d876b5580f7299c WatchSource:0}: Error finding container 567f0b151e82574b6353852636e3cb2b7534941899a86e459d876b5580f7299c: Status 404 returned error can't find the container with id 567f0b151e82574b6353852636e3cb2b7534941899a86e459d876b5580f7299c Sep 15 12:57:35.926780 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.926741 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" event={"ID":"a3f76539-5719-467b-b2be-47f030ca9160","Type":"ContainerStarted","Data":"4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29"} Sep 15 12:57:35.926975 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.926789 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" event={"ID":"a3f76539-5719-467b-b2be-47f030ca9160","Type":"ContainerStarted","Data":"567f0b151e82574b6353852636e3cb2b7534941899a86e459d876b5580f7299c"} Sep 15 12:57:35.926975 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.926822 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:35.943642 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:35.943587 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" podStartSLOduration=0.943569195 podStartE2EDuration="943.569195ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:57:35.941598318 +0000 UTC m=+1937.296444644" watchObservedRunningTime="2026-09-15 12:57:35.943569195 +0000 UTC m=+1937.298415520" Sep 15 12:57:46.934168 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:46.934131 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 12:57:47.680258 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:47.679218 2550 patch_prober.go:28] interesting pod/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": read tcp 10.132.0.2:41022->10.132.0.88:15021: read: connection reset by peer" start-of-body= Sep 15 12:57:47.680258 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:47.679293 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": read tcp 10.132.0.2:41022->10.132.0.88:15021: read: connection reset by peer" Sep 15 12:57:48.003287 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:48.003250 2550 generic.go:356] "Generic (PLEG): container finished" podID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerID="083f04de6a189d2b595c28dfa3c141fbe87d9766c7c4c8607dcf3cbb04db31f6" exitCode=0 Sep 15 12:57:48.003761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:48.003321 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" event={"ID":"6b316189-29eb-4bc7-95eb-d0ea6325c2a0","Type":"ContainerDied","Data":"083f04de6a189d2b595c28dfa3c141fbe87d9766c7c4c8607dcf3cbb04db31f6"} Sep 15 12:57:48.003761 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:48.003742 2550 scope.go:117] "RemoveContainer" containerID="083f04de6a189d2b595c28dfa3c141fbe87d9766c7c4c8607dcf3cbb04db31f6" Sep 15 12:57:48.004983 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:48.004941 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:48.005083 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:48.005047 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:48.005147 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:48.005123 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:49.011102 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:49.011062 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" event={"ID":"6b316189-29eb-4bc7-95eb-d0ea6325c2a0","Type":"ContainerStarted","Data":"236aa0b470c7285cc6168492e98c9ab926205c92fc285f187656f5a65948b65e"} Sep 15 12:57:49.682229 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:49.682198 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:49.682461 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:49.682447 2550 patch_prober.go:28] interesting pod/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 12:57:49.682525 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:49.682488 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 12:57:50.682930 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:50.682891 2550 patch_prober.go:28] interesting pod/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 12:57:50.683297 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:50.682947 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 12:57:51.683246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:51.683206 2550 patch_prober.go:28] interesting pod/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 12:57:51.683689 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:51.683263 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 12:57:52.683302 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:52.683264 2550 patch_prober.go:28] interesting pod/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 12:57:52.683713 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:52.683323 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 12:57:53.682541 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:53.682503 2550 patch_prober.go:28] interesting pod/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 15 12:57:53.682719 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:53.682576 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 15 12:57:54.685925 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:54.685892 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:54.686326 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:54.686141 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:54.686936 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:54.686917 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 12:57:58.817804 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.817769 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w"] Sep 15 12:57:58.821697 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.821673 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.824056 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.824024 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cd72a9-openshift-default-dockercfg-mf4jj\"" Sep 15 12:57:58.833334 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.833311 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w"] Sep 15 12:57:58.940255 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940216 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-data\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940478 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940273 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-credential-socket\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940478 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940338 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-token\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940478 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940418 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23ab45e-850b-4206-8cc9-19add3dd35f6-istiod-ca-cert\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940478 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940471 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-podinfo\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940508 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ktf\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-kube-api-access-d9ktf\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940619 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-socket\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940681 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-envoy\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:58.940727 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:58.940710 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-certs\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042039 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042007 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-token\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042039 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042041 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23ab45e-850b-4206-8cc9-19add3dd35f6-istiod-ca-cert\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042068 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-podinfo\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042148 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ktf\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-kube-api-access-d9ktf\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-socket\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042316 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042290 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-envoy\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042322 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-certs\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042408 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-data\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042590 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042444 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-credential-socket\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042737 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042708 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-socket\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042825 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042801 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-data\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042887 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042826 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-certs\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042947 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042902 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23ab45e-850b-4206-8cc9-19add3dd35f6-istiod-ca-cert\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.042947 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.042915 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-credential-socket\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.044514 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.044488 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-envoy\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.044743 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.044724 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-podinfo\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.049895 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.049870 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-token\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.050401 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.050358 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ktf\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-kube-api-access-d9ktf\") pod \"e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.134556 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.134471 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:57:59.412924 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.412892 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w"] Sep 15 12:57:59.414048 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:57:59.414019 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23ab45e_850b_4206_8cc9_19add3dd35f6.slice/crio-20a0922c805d6a113f4f092cab016f537053892a147aa2f698aeb1297fe1cf6e WatchSource:0}: Error finding container 20a0922c805d6a113f4f092cab016f537053892a147aa2f698aeb1297fe1cf6e: Status 404 returned error can't find the container with id 20a0922c805d6a113f4f092cab016f537053892a147aa2f698aeb1297fe1cf6e Sep 15 12:57:59.416165 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.416133 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:59.416256 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.416209 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:57:59.416319 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:57:59.416254 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:58:00.076374 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:00.076337 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" event={"ID":"a23ab45e-850b-4206-8cc9-19add3dd35f6","Type":"ContainerStarted","Data":"1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be"} Sep 15 12:58:00.076768 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:00.076405 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" event={"ID":"a23ab45e-850b-4206-8cc9-19add3dd35f6","Type":"ContainerStarted","Data":"20a0922c805d6a113f4f092cab016f537053892a147aa2f698aeb1297fe1cf6e"} Sep 15 12:58:00.096047 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:00.095988 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" podStartSLOduration=2.095971878 podStartE2EDuration="2.095971878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:58:00.093697658 +0000 UTC m=+1961.448543980" watchObservedRunningTime="2026-09-15 12:58:00.095971878 +0000 UTC m=+1961.450818202" Sep 15 12:58:00.135650 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:00.135619 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:00.140586 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:00.140561 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:01.082197 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:01.082168 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:01.083290 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:01.083269 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:19.685803 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:19.685768 2550 scope.go:117] "RemoveContainer" containerID="eacddb07d11780718065ef8b726f6dc86ebadcb87213f8c69701eb5c42d866e3" Sep 15 12:58:19.695308 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:19.695278 2550 scope.go:117] "RemoveContainer" containerID="ebfcbad75f7e37ca936aea28809b87afa2ca48cc3e96ac3f4060e05b6a5876a2" Sep 15 12:58:20.208899 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:20.208864 2550 generic.go:356] "Generic (PLEG): container finished" podID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerID="1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be" exitCode=0 Sep 15 12:58:20.209101 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:20.208939 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" event={"ID":"a23ab45e-850b-4206-8cc9-19add3dd35f6","Type":"ContainerDied","Data":"1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be"} Sep 15 12:58:20.209373 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:20.209358 2550 scope.go:117] "RemoveContainer" containerID="1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be" Sep 15 12:58:20.210591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:20.210553 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:58:20.210694 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:20.210634 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:58:20.217619 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:20.211176 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:58:21.217116 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:21.217083 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" event={"ID":"a23ab45e-850b-4206-8cc9-19add3dd35f6","Type":"ContainerStarted","Data":"2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380"} Sep 15 12:58:22.135299 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:22.135258 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:22.135531 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:22.135406 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 15 12:58:22.135531 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:22.135452 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 15 12:58:23.134976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:23.134928 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 15 12:58:23.135487 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:23.135004 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 15 12:58:24.136053 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:24.136006 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 15 12:58:24.136636 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:24.136083 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 15 12:58:25.135559 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:25.135510 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 15 12:58:25.135788 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:25.135611 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 15 12:58:26.135503 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:26.135464 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 15 12:58:26.135871 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:26.135522 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 15 12:58:27.138740 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:27.138709 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:27.139216 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:27.138996 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:27.139674 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:27.139656 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 12:58:32.419617 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:58:32.419587 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-v9pnn_107e94e7-82fd-4fda-a35e-28b1d35c5b2b/payload-processing/0.log" Sep 15 12:59:19.719343 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:19.719309 2550 scope.go:117] "RemoveContainer" containerID="561178eaeb57ad3efa1b8a88aae94e9523f228a3b5273de204e8dd7ca8e115d6" Sep 15 12:59:19.727606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:19.727583 2550 scope.go:117] "RemoveContainer" containerID="777ed2022b98f72f373f4f6c902ca21b7e98e54a4aaf36947b1034d2e4239e81" Sep 15 12:59:23.826977 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:23.826932 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-v9pnn_107e94e7-82fd-4fda-a35e-28b1d35c5b2b/payload-processing/0.log" Sep 15 12:59:24.559289 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.559257 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr"] Sep 15 12:59:24.562928 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.562909 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.565252 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.565233 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-85cea9-openshift-default-dockercfg-65gnm\"" Sep 15 12:59:24.573940 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.573913 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr"] Sep 15 12:59:24.708684 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708649 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqgd\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-kube-api-access-zkqgd\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.708684 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708686 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-socket\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.708901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708711 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-data\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.708901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708735 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-credential-socket\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.708901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708763 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-podinfo\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.708901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708780 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2140e80-7018-4cad-ac47-058943f7e7cc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.708901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708857 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-token\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.708901 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708891 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-certs\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.709125 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.708939 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-envoy\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810367 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810284 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqgd\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-kube-api-access-zkqgd\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810367 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810332 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-socket\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810370 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-data\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-credential-socket\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810463 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-podinfo\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810488 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2140e80-7018-4cad-ac47-058943f7e7cc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810542 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-token\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810606 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810577 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-certs\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810871 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810626 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-envoy\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810871 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810727 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-socket\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810871 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810827 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-credential-socket\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.810994 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.810970 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-certs\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.811124 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.811103 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-data\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.811222 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.811205 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2140e80-7018-4cad-ac47-058943f7e7cc-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.812916 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.812894 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-envoy\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.813181 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.813161 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-podinfo\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.817714 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.817692 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-token\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.818000 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.817980 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqgd\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-kube-api-access-zkqgd\") pod \"e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:24.873616 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:24.873559 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:25.162246 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.162205 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:59:25.162419 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.162285 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:59:25.162419 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.162328 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:59:25.163322 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.163297 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr"] Sep 15 12:59:25.568369 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.568333 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" event={"ID":"f2140e80-7018-4cad-ac47-058943f7e7cc","Type":"ContainerStarted","Data":"cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43"} Sep 15 12:59:25.568580 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.568402 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" event={"ID":"f2140e80-7018-4cad-ac47-058943f7e7cc","Type":"ContainerStarted","Data":"6d28af4c44968f1df8c9f65ef3eac1da9e9e5d8c05c340deda8f7060dc5f401e"} Sep 15 12:59:25.586710 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.586661 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podStartSLOduration=1.586645942 podStartE2EDuration="1.586645942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:59:25.58528004 +0000 UTC m=+2046.940126364" watchObservedRunningTime="2026-09-15 12:59:25.586645942 +0000 UTC m=+2046.941492264" Sep 15 12:59:25.873961 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:25.873877 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:26.879664 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:26.879637 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:27.579287 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:27.579256 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:27.580167 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:27.580149 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:31.196707 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.196664 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh"] Sep 15 12:59:31.209284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.209248 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.209568 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.209545 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh"] Sep 15 12:59:31.211443 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.211408 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-85cea9-dockercfg-ljxbj\"" Sep 15 12:59:31.211858 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.211479 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-85cea9\"" Sep 15 12:59:31.270059 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.270018 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac55bda-36a1-41db-bd70-ebadde7c9635-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.270364 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.270340 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrnr\" (UniqueName: \"kubernetes.io/projected/5ac55bda-36a1-41db-bd70-ebadde7c9635-kube-api-access-tnrnr\") pod \"payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.371099 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.371063 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac55bda-36a1-41db-bd70-ebadde7c9635-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.371278 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.371157 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrnr\" (UniqueName: \"kubernetes.io/projected/5ac55bda-36a1-41db-bd70-ebadde7c9635-kube-api-access-tnrnr\") pod \"payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.371806 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.371786 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac55bda-36a1-41db-bd70-ebadde7c9635-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.378651 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.378626 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrnr\" (UniqueName: \"kubernetes.io/projected/5ac55bda-36a1-41db-bd70-ebadde7c9635-kube-api-access-tnrnr\") pod \"payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.522976 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.522945 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:31.796205 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:31.796178 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh"] Sep 15 12:59:31.797785 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:59:31.797761 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac55bda_36a1_41db_bd70_ebadde7c9635.slice/crio-6955a2f8aed9b57198649395e8c279b2b24fe0f9fac26b513e4495fa2187ea1a WatchSource:0}: Error finding container 6955a2f8aed9b57198649395e8c279b2b24fe0f9fac26b513e4495fa2187ea1a: Status 404 returned error can't find the container with id 6955a2f8aed9b57198649395e8c279b2b24fe0f9fac26b513e4495fa2187ea1a Sep 15 12:59:32.608177 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:32.608142 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" event={"ID":"5ac55bda-36a1-41db-bd70-ebadde7c9635","Type":"ContainerStarted","Data":"7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286"} Sep 15 12:59:32.608580 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:32.608203 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" event={"ID":"5ac55bda-36a1-41db-bd70-ebadde7c9635","Type":"ContainerStarted","Data":"6955a2f8aed9b57198649395e8c279b2b24fe0f9fac26b513e4495fa2187ea1a"} Sep 15 12:59:32.608580 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:32.608250 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:32.632695 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:32.632641 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" podStartSLOduration=1.632621584 podStartE2EDuration="1.632621584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:59:32.63081047 +0000 UTC m=+2053.985656792" watchObservedRunningTime="2026-09-15 12:59:32.632621584 +0000 UTC m=+2053.987467908" Sep 15 12:59:43.619348 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:43.619274 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:48.705579 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:48.705540 2550 generic.go:356] "Generic (PLEG): container finished" podID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerID="cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43" exitCode=0 Sep 15 12:59:48.706123 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:48.705624 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" event={"ID":"f2140e80-7018-4cad-ac47-058943f7e7cc","Type":"ContainerDied","Data":"cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43"} Sep 15 12:59:48.706231 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:48.706213 2550 scope.go:117] "RemoveContainer" containerID="cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43" Sep 15 12:59:48.707405 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:48.707362 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:59:48.707479 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:48.707445 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:59:48.707517 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:48.707476 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"} Sep 15 12:59:49.713875 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:49.713843 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" event={"ID":"f2140e80-7018-4cad-ac47-058943f7e7cc","Type":"ContainerStarted","Data":"542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74"} Sep 15 12:59:49.874687 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:49.874652 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:49.874910 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:49.874888 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 15 12:59:49.874957 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:49.874935 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 15 12:59:50.874867 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:50.874829 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 15 12:59:50.875284 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:50.874901 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 15 12:59:51.874888 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:51.874849 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 15 12:59:51.875516 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:51.875482 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 15 12:59:52.874938 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:52.874896 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 15 12:59:52.875343 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:52.874955 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 15 12:59:53.168618 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.168531 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj"] Sep 15 12:59:53.171995 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.171979 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" Sep 15 12:59:53.174267 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.174249 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 12:59:53.174398 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.174283 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 12:59:53.180301 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.180275 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj"] Sep 15 12:59:53.297436 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.297402 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj\" (UID: \"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" Sep 15 12:59:53.398334 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.398298 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj\" (UID: \"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" Sep 15 12:59:53.399018 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.398996 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj\" (UID: \"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" Sep 15 12:59:53.485488 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.485405 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" Sep 15 12:59:53.772882 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.772853 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj"] Sep 15 12:59:53.774344 ip-10-0-137-242 kubenswrapper[2550]: W0915 12:59:53.774320 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0dc84c_d5e2_4c1a_8011_06e785ed5f7d.slice/crio-2e98188d1370851d69a4fd77add37dbf5d0ac60a4b8e82d55fdbb8aaa4f6438f WatchSource:0}: Error finding container 2e98188d1370851d69a4fd77add37dbf5d0ac60a4b8e82d55fdbb8aaa4f6438f: Status 404 returned error can't find the container with id 2e98188d1370851d69a4fd77add37dbf5d0ac60a4b8e82d55fdbb8aaa4f6438f Sep 15 12:59:53.874101 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.874067 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 15 12:59:53.874269 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:53.874126 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 15 12:59:54.758645 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:54.758602 2550 generic.go:356] "Generic (PLEG): container finished" podID="5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d" containerID="ac7749b274e8b843d517128a7e2d35d7d05bcbbff30acc6f2f84cb78751f2c87" exitCode=0 Sep 15 12:59:54.759214 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:54.758748 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" event={"ID":"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d","Type":"ContainerDied","Data":"ac7749b274e8b843d517128a7e2d35d7d05bcbbff30acc6f2f84cb78751f2c87"} Sep 15 12:59:54.759360 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:54.759342 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" event={"ID":"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d","Type":"ContainerStarted","Data":"2e98188d1370851d69a4fd77add37dbf5d0ac60a4b8e82d55fdbb8aaa4f6438f"} Sep 15 12:59:54.874822 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:54.874784 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 15 12:59:54.874998 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:54.874859 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 15 12:59:55.878033 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:55.878002 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:55.878510 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:55.878406 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:55.879310 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:55.879285 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 12:59:55.904554 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:55.904531 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" Sep 15 12:59:56.024506 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:56.024412 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d-maas-api-service-ca\") pod \"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d\" (UID: \"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d\") " Sep 15 12:59:56.024758 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:56.024732 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d" (UID: "5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:59:56.125722 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:56.125682 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:59:56.771327 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:56.771296 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" Sep 15 12:59:56.771505 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:56.771295 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj" event={"ID":"5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d","Type":"ContainerDied","Data":"2e98188d1370851d69a4fd77add37dbf5d0ac60a4b8e82d55fdbb8aaa4f6438f"} Sep 15 12:59:56.771505 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:56.771426 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e98188d1370851d69a4fd77add37dbf5d0ac60a4b8e82d55fdbb8aaa4f6438f" Sep 15 12:59:59.398578 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.398537 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh"] Sep 15 12:59:59.399005 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.398782 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" podUID="5ac55bda-36a1-41db-bd70-ebadde7c9635" containerName="payload-processing" containerID="cri-o://7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286" gracePeriod=30 Sep 15 12:59:59.564009 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.563970 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj"] Sep 15 12:59:59.566077 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.566049 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-85cea9-m4jpj"] Sep 15 12:59:59.660671 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.660602 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:59.761006 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.760963 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac55bda-36a1-41db-bd70-ebadde7c9635-plugins-config-volume\") pod \"5ac55bda-36a1-41db-bd70-ebadde7c9635\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " Sep 15 12:59:59.761777 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.761666 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac55bda-36a1-41db-bd70-ebadde7c9635-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ac55bda-36a1-41db-bd70-ebadde7c9635" (UID: "5ac55bda-36a1-41db-bd70-ebadde7c9635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:59:59.762039 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.762008 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnrnr\" (UniqueName: \"kubernetes.io/projected/5ac55bda-36a1-41db-bd70-ebadde7c9635-kube-api-access-tnrnr\") pod \"5ac55bda-36a1-41db-bd70-ebadde7c9635\" (UID: \"5ac55bda-36a1-41db-bd70-ebadde7c9635\") " Sep 15 12:59:59.762414 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.762394 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac55bda-36a1-41db-bd70-ebadde7c9635-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 12:59:59.765723 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.765698 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac55bda-36a1-41db-bd70-ebadde7c9635-kube-api-access-tnrnr" (OuterVolumeSpecName: "kube-api-access-tnrnr") pod "5ac55bda-36a1-41db-bd70-ebadde7c9635" (UID: "5ac55bda-36a1-41db-bd70-ebadde7c9635"). InnerVolumeSpecName "kube-api-access-tnrnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:59:59.794747 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.794715 2550 generic.go:356] "Generic (PLEG): container finished" podID="5ac55bda-36a1-41db-bd70-ebadde7c9635" containerID="7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286" exitCode=0 Sep 15 12:59:59.794908 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.794763 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" Sep 15 12:59:59.794908 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.794808 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" event={"ID":"5ac55bda-36a1-41db-bd70-ebadde7c9635","Type":"ContainerDied","Data":"7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286"} Sep 15 12:59:59.794908 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.794860 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh" event={"ID":"5ac55bda-36a1-41db-bd70-ebadde7c9635","Type":"ContainerDied","Data":"6955a2f8aed9b57198649395e8c279b2b24fe0f9fac26b513e4495fa2187ea1a"} Sep 15 12:59:59.794908 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.794884 2550 scope.go:117] "RemoveContainer" containerID="7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286" Sep 15 12:59:59.803784 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.803767 2550 scope.go:117] "RemoveContainer" containerID="7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286" Sep 15 12:59:59.804037 ip-10-0-137-242 kubenswrapper[2550]: E0915 12:59:59.804018 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286\": container with ID starting with 7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286 not found: ID does not exist" containerID="7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286" Sep 15 12:59:59.804088 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.804045 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286"} err="failed to get container status \"7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286\": rpc error: code = NotFound desc = could not find container \"7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286\": container with ID starting with 7a8176e37c084e1f4202396de452d23ca712e38207c1aab2f21169e1990c8286 not found: ID does not exist" Sep 15 12:59:59.815143 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.815117 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh"] Sep 15 12:59:59.817254 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.817234 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-85cea9-5f78f6bc89-jnfmh"] Sep 15 12:59:59.863591 ip-10-0-137-242 kubenswrapper[2550]: I0915 12:59:59.863561 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrnr\" (UniqueName: \"kubernetes.io/projected/5ac55bda-36a1-41db-bd70-ebadde7c9635-kube-api-access-tnrnr\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:00.125855 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.125817 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss"] Sep 15 13:00:00.126250 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.126237 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d" containerName="revoke-keys" Sep 15 13:00:00.126306 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.126251 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d" containerName="revoke-keys" Sep 15 13:00:00.126306 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.126279 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac55bda-36a1-41db-bd70-ebadde7c9635" containerName="payload-processing" Sep 15 13:00:00.126306 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.126286 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac55bda-36a1-41db-bd70-ebadde7c9635" containerName="payload-processing" Sep 15 13:00:00.126433 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.126353 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac55bda-36a1-41db-bd70-ebadde7c9635" containerName="payload-processing" Sep 15 13:00:00.126433 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.126362 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d" containerName="revoke-keys" Sep 15 13:00:00.130873 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.130846 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" Sep 15 13:00:00.132979 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.132959 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 13:00:00.133319 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.133299 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst"] Sep 15 13:00:00.137046 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.137023 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj"] Sep 15 13:00:00.137156 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.137108 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" Sep 15 13:00:00.141021 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.140999 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss"] Sep 15 13:00:00.141133 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.141116 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" Sep 15 13:00:00.146357 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.145810 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst"] Sep 15 13:00:00.146357 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.146092 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" Sep 15 13:00:00.153033 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.152800 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj"] Sep 15 13:00:00.157503 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.157481 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" Sep 15 13:00:00.179015 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.178970 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" Sep 15 13:00:00.428489 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.428465 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss"] Sep 15 13:00:00.430287 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:00:00.430257 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9445b8d_c0a8_4331_b658_2f640705c8dd.slice/crio-999a357f5272fae316390ec866a35ebc2dfdd4984af018f62af201e8244da01d WatchSource:0}: Error finding container 999a357f5272fae316390ec866a35ebc2dfdd4984af018f62af201e8244da01d: Status 404 returned error can't find the container with id 999a357f5272fae316390ec866a35ebc2dfdd4984af018f62af201e8244da01d Sep 15 13:00:00.440656 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.440638 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst"] Sep 15 13:00:00.444040 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:00:00.444020 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba54e8e_96a2_4b9e_a773_eccd2c761195.slice/crio-5bcf38998ba25f7ab0541d8f1fab42b3953fd7e2e3c0ed2157630106af3380c3 WatchSource:0}: Error finding container 5bcf38998ba25f7ab0541d8f1fab42b3953fd7e2e3c0ed2157630106af3380c3: Status 404 returned error can't find the container with id 5bcf38998ba25f7ab0541d8f1fab42b3953fd7e2e3c0ed2157630106af3380c3 Sep 15 13:00:00.465879 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.465856 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj"] Sep 15 13:00:00.468268 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:00:00.468244 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cb9325_41d4_4920_b35b_a4b6cf8f0136.slice/crio-71cbfc6e4898d9f0692563fcb3a0ec697c9c624331d9463705a74b729fb7ee59 WatchSource:0}: Error finding container 71cbfc6e4898d9f0692563fcb3a0ec697c9c624331d9463705a74b729fb7ee59: Status 404 returned error can't find the container with id 71cbfc6e4898d9f0692563fcb3a0ec697c9c624331d9463705a74b729fb7ee59 Sep 15 13:00:00.801863 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.801823 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" event={"ID":"e9445b8d-c0a8-4331-b658-2f640705c8dd","Type":"ContainerStarted","Data":"9e2bef73d6e491aad6a6eb97dc562e85160dfcdf4e557aef80a161f063302345"} Sep 15 13:00:00.802040 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.801880 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" event={"ID":"e9445b8d-c0a8-4331-b658-2f640705c8dd","Type":"ContainerStarted","Data":"999a357f5272fae316390ec866a35ebc2dfdd4984af018f62af201e8244da01d"} Sep 15 13:00:00.803295 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.803268 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" event={"ID":"bba54e8e-96a2-4b9e-a773-eccd2c761195","Type":"ContainerStarted","Data":"49b1aafa78729cf0e7f35979acc2d20fe0c93e3f2d86ff3f8d8191e1175fb0b1"} Sep 15 13:00:00.803435 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.803310 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" event={"ID":"bba54e8e-96a2-4b9e-a773-eccd2c761195","Type":"ContainerStarted","Data":"5bcf38998ba25f7ab0541d8f1fab42b3953fd7e2e3c0ed2157630106af3380c3"} Sep 15 13:00:00.805649 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.805630 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" event={"ID":"43cb9325-41d4-4920-b35b-a4b6cf8f0136","Type":"ContainerStarted","Data":"adb947942bc80180227615ea0642f9450f20a48851db9c3f8e18a4092cd9490d"} Sep 15 13:00:00.805744 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.805663 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" event={"ID":"43cb9325-41d4-4920-b35b-a4b6cf8f0136","Type":"ContainerStarted","Data":"71cbfc6e4898d9f0692563fcb3a0ec697c9c624331d9463705a74b729fb7ee59"} Sep 15 13:00:00.826893 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.824475 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" podStartSLOduration=0.824456884 podStartE2EDuration="824.456884ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:00:00.818760775 +0000 UTC m=+2082.173607114" watchObservedRunningTime="2026-09-15 13:00:00.824456884 +0000 UTC m=+2082.179303207" Sep 15 13:00:00.840106 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.839989 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" podStartSLOduration=0.839963493 podStartE2EDuration="839.963493ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:00:00.838318057 +0000 UTC m=+2082.193164380" watchObservedRunningTime="2026-09-15 13:00:00.839963493 +0000 UTC m=+2082.194809887" Sep 15 13:00:00.857456 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:00.857408 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" podStartSLOduration=0.857393348 podStartE2EDuration="857.393348ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:00:00.85566074 +0000 UTC m=+2082.210507064" watchObservedRunningTime="2026-09-15 13:00:00.857393348 +0000 UTC m=+2082.212239662" Sep 15 13:00:01.154729 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.154645 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac55bda-36a1-41db-bd70-ebadde7c9635" path="/var/lib/kubelet/pods/5ac55bda-36a1-41db-bd70-ebadde7c9635/volumes" Sep 15 13:00:01.155012 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.155000 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d" path="/var/lib/kubelet/pods/5c0dc84c-d5e2-4c1a-8011-06e785ed5f7d/volumes" Sep 15 13:00:01.815229 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.815190 2550 generic.go:356] "Generic (PLEG): container finished" podID="e9445b8d-c0a8-4331-b658-2f640705c8dd" containerID="9e2bef73d6e491aad6a6eb97dc562e85160dfcdf4e557aef80a161f063302345" exitCode=0 Sep 15 13:00:01.815664 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.815268 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" event={"ID":"e9445b8d-c0a8-4331-b658-2f640705c8dd","Type":"ContainerDied","Data":"9e2bef73d6e491aad6a6eb97dc562e85160dfcdf4e557aef80a161f063302345"} Sep 15 13:00:01.817010 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.816988 2550 generic.go:356] "Generic (PLEG): container finished" podID="bba54e8e-96a2-4b9e-a773-eccd2c761195" containerID="49b1aafa78729cf0e7f35979acc2d20fe0c93e3f2d86ff3f8d8191e1175fb0b1" exitCode=0 Sep 15 13:00:01.817124 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.817067 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" event={"ID":"bba54e8e-96a2-4b9e-a773-eccd2c761195","Type":"ContainerDied","Data":"49b1aafa78729cf0e7f35979acc2d20fe0c93e3f2d86ff3f8d8191e1175fb0b1"} Sep 15 13:00:01.818550 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.818530 2550 generic.go:356] "Generic (PLEG): container finished" podID="43cb9325-41d4-4920-b35b-a4b6cf8f0136" containerID="adb947942bc80180227615ea0642f9450f20a48851db9c3f8e18a4092cd9490d" exitCode=0 Sep 15 13:00:01.818642 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:01.818568 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" event={"ID":"43cb9325-41d4-4920-b35b-a4b6cf8f0136","Type":"ContainerDied","Data":"adb947942bc80180227615ea0642f9450f20a48851db9c3f8e18a4092cd9490d"} Sep 15 13:00:02.976540 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:02.976512 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" Sep 15 13:00:03.027454 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.027430 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" Sep 15 13:00:03.030413 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.030355 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" Sep 15 13:00:03.830994 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.830961 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" Sep 15 13:00:03.831177 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.830997 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj" event={"ID":"43cb9325-41d4-4920-b35b-a4b6cf8f0136","Type":"ContainerDied","Data":"71cbfc6e4898d9f0692563fcb3a0ec697c9c624331d9463705a74b729fb7ee59"} Sep 15 13:00:03.831177 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.831060 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71cbfc6e4898d9f0692563fcb3a0ec697c9c624331d9463705a74b729fb7ee59" Sep 15 13:00:03.832467 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.832446 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" Sep 15 13:00:03.832604 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.832471 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss" event={"ID":"e9445b8d-c0a8-4331-b658-2f640705c8dd","Type":"ContainerDied","Data":"999a357f5272fae316390ec866a35ebc2dfdd4984af018f62af201e8244da01d"} Sep 15 13:00:03.832604 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.832519 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="999a357f5272fae316390ec866a35ebc2dfdd4984af018f62af201e8244da01d" Sep 15 13:00:03.833939 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.833922 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" Sep 15 13:00:03.834061 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.833958 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst" event={"ID":"bba54e8e-96a2-4b9e-a773-eccd2c761195","Type":"ContainerDied","Data":"5bcf38998ba25f7ab0541d8f1fab42b3953fd7e2e3c0ed2157630106af3380c3"} Sep 15 13:00:03.834061 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:03.833994 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bcf38998ba25f7ab0541d8f1fab42b3953fd7e2e3c0ed2157630106af3380c3" Sep 15 13:00:05.826292 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:05.826245 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr"] Sep 15 13:00:05.827136 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:05.827099 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" containerID="cri-o://542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74" gracePeriod=30 Sep 15 13:00:09.409886 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.409852 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd"] Sep 15 13:00:09.410287 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410266 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cb9325-41d4-4920-b35b-a4b6cf8f0136" containerName="cleanup" Sep 15 13:00:09.410339 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410290 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cb9325-41d4-4920-b35b-a4b6cf8f0136" containerName="cleanup" Sep 15 13:00:09.410339 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410301 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba54e8e-96a2-4b9e-a773-eccd2c761195" containerName="cleanup" Sep 15 13:00:09.410339 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410306 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba54e8e-96a2-4b9e-a773-eccd2c761195" containerName="cleanup" Sep 15 13:00:09.410339 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410319 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9445b8d-c0a8-4331-b658-2f640705c8dd" containerName="cleanup" Sep 15 13:00:09.410339 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410327 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9445b8d-c0a8-4331-b658-2f640705c8dd" containerName="cleanup" Sep 15 13:00:09.410567 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410424 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba54e8e-96a2-4b9e-a773-eccd2c761195" containerName="cleanup" Sep 15 13:00:09.410567 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410438 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9445b8d-c0a8-4331-b658-2f640705c8dd" containerName="cleanup" Sep 15 13:00:09.410567 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.410445 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cb9325-41d4-4920-b35b-a4b6cf8f0136" containerName="cleanup" Sep 15 13:00:09.413698 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.413683 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" Sep 15 13:00:09.415732 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.415704 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 13:00:09.415861 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.415744 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 13:00:09.419721 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.419635 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd"] Sep 15 13:00:09.570962 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.570925 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6507dd30-6419-4eac-8501-4789264c9c88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd\" (UID: \"6507dd30-6419-4eac-8501-4789264c9c88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" Sep 15 13:00:09.672566 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.672478 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6507dd30-6419-4eac-8501-4789264c9c88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd\" (UID: \"6507dd30-6419-4eac-8501-4789264c9c88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" Sep 15 13:00:09.673106 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.673085 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6507dd30-6419-4eac-8501-4789264c9c88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd\" (UID: \"6507dd30-6419-4eac-8501-4789264c9c88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" Sep 15 13:00:09.727409 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.727352 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" Sep 15 13:00:09.998082 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:09.998058 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd"] Sep 15 13:00:09.999058 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:00:09.999034 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6507dd30_6419_4eac_8501_4789264c9c88.slice/crio-0b2fd13625d8986580f687e263b720305557153181e3ea8859d69ec9ac203746 WatchSource:0}: Error finding container 0b2fd13625d8986580f687e263b720305557153181e3ea8859d69ec9ac203746: Status 404 returned error can't find the container with id 0b2fd13625d8986580f687e263b720305557153181e3ea8859d69ec9ac203746 Sep 15 13:00:10.873492 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:10.873454 2550 generic.go:356] "Generic (PLEG): container finished" podID="6507dd30-6419-4eac-8501-4789264c9c88" containerID="ef14277feebfdbf2036c4352b94b97d2ec2dc675f3fd2128d731e212cd86822f" exitCode=0 Sep 15 13:00:10.873845 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:10.873500 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" event={"ID":"6507dd30-6419-4eac-8501-4789264c9c88","Type":"ContainerDied","Data":"ef14277feebfdbf2036c4352b94b97d2ec2dc675f3fd2128d731e212cd86822f"} Sep 15 13:00:10.873845 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:10.873558 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" event={"ID":"6507dd30-6419-4eac-8501-4789264c9c88","Type":"ContainerStarted","Data":"0b2fd13625d8986580f687e263b720305557153181e3ea8859d69ec9ac203746"} Sep 15 13:00:10.879671 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:10.879647 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 15 13:00:10.879787 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:10.879694 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 15 13:00:11.105460 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.105427 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 13:00:11.286326 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286295 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-data\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286561 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286334 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-podinfo\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286561 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286358 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-certs\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286561 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286435 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2140e80-7018-4cad-ac47-058943f7e7cc-istiod-ca-cert\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286561 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286454 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-envoy\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286561 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286471 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-socket\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286561 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286504 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-token\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286561 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286560 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkqgd\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-kube-api-access-zkqgd\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286948 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286613 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-credential-socket\") pod \"f2140e80-7018-4cad-ac47-058943f7e7cc\" (UID: \"f2140e80-7018-4cad-ac47-058943f7e7cc\") " Sep 15 13:00:11.286948 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286655 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:00:11.286948 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286807 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:00:11.286948 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286899 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2140e80-7018-4cad-ac47-058943f7e7cc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:00:11.287161 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286958 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.287161 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.286977 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.287161 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.287044 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:00:11.287161 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.287061 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:00:11.288644 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.288620 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 13:00:11.288807 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.288778 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:00:11.288918 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.288813 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:00:11.289028 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.289011 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-kube-api-access-zkqgd" (OuterVolumeSpecName: "kube-api-access-zkqgd") pod "f2140e80-7018-4cad-ac47-058943f7e7cc" (UID: "f2140e80-7018-4cad-ac47-058943f7e7cc"). InnerVolumeSpecName "kube-api-access-zkqgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:00:11.388020 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.387931 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2140e80-7018-4cad-ac47-058943f7e7cc-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.388020 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.387963 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.388020 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.387971 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.388020 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.387979 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqgd\" (UniqueName: \"kubernetes.io/projected/f2140e80-7018-4cad-ac47-058943f7e7cc-kube-api-access-zkqgd\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.388020 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.387990 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.388020 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.387998 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2140e80-7018-4cad-ac47-058943f7e7cc-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.388020 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.388006 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2140e80-7018-4cad-ac47-058943f7e7cc-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:11.880755 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.880719 2550 generic.go:356] "Generic (PLEG): container finished" podID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerID="542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74" exitCode=0 Sep 15 13:00:11.881169 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.880772 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" Sep 15 13:00:11.881169 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.880806 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" event={"ID":"f2140e80-7018-4cad-ac47-058943f7e7cc","Type":"ContainerDied","Data":"542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74"} Sep 15 13:00:11.881169 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.880865 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr" event={"ID":"f2140e80-7018-4cad-ac47-058943f7e7cc","Type":"ContainerDied","Data":"6d28af4c44968f1df8c9f65ef3eac1da9e9e5d8c05c340deda8f7060dc5f401e"} Sep 15 13:00:11.881169 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.880892 2550 scope.go:117] "RemoveContainer" containerID="542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74" Sep 15 13:00:11.891435 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.891416 2550 scope.go:117] "RemoveContainer" containerID="cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43" Sep 15 13:00:11.900229 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.900210 2550 scope.go:117] "RemoveContainer" containerID="542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74" Sep 15 13:00:11.900587 ip-10-0-137-242 kubenswrapper[2550]: E0915 13:00:11.900565 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74\": container with ID starting with 542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74 not found: ID does not exist" containerID="542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74" Sep 15 13:00:11.900652 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.900595 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74"} err="failed to get container status \"542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74\": rpc error: code = NotFound desc = could not find container \"542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74\": container with ID starting with 542fefdd6c5c935cab3ae1e86254fe5360c33b1be7e819ef81517677f9ab9b74 not found: ID does not exist" Sep 15 13:00:11.900652 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.900612 2550 scope.go:117] "RemoveContainer" containerID="cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43" Sep 15 13:00:11.900944 ip-10-0-137-242 kubenswrapper[2550]: E0915 13:00:11.900913 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43\": container with ID starting with cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43 not found: ID does not exist" containerID="cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43" Sep 15 13:00:11.901071 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.900949 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43"} err="failed to get container status \"cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43\": rpc error: code = NotFound desc = could not find container \"cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43\": container with ID starting with cba3ce9d1122fb5d69286c1bb694220d8a6491f6e71aa5c7412b9fd10b089a43 not found: ID does not exist" Sep 15 13:00:11.904019 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.903998 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr"] Sep 15 13:00:11.908212 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:11.908190 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85cea9-openshift-default-65bd5cf466-s78cr"] Sep 15 13:00:12.021998 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:12.021972 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" Sep 15 13:00:12.094478 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:12.094445 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6507dd30-6419-4eac-8501-4789264c9c88-maas-api-service-ca\") pod \"6507dd30-6419-4eac-8501-4789264c9c88\" (UID: \"6507dd30-6419-4eac-8501-4789264c9c88\") " Sep 15 13:00:12.094798 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:12.094775 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6507dd30-6419-4eac-8501-4789264c9c88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6507dd30-6419-4eac-8501-4789264c9c88" (UID: "6507dd30-6419-4eac-8501-4789264c9c88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:00:12.195613 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:12.195520 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6507dd30-6419-4eac-8501-4789264c9c88-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:12.889284 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:12.889246 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" event={"ID":"6507dd30-6419-4eac-8501-4789264c9c88","Type":"ContainerDied","Data":"0b2fd13625d8986580f687e263b720305557153181e3ea8859d69ec9ac203746"} Sep 15 13:00:12.889681 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:12.889302 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b2fd13625d8986580f687e263b720305557153181e3ea8859d69ec9ac203746" Sep 15 13:00:12.889681 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:12.889253 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd" Sep 15 13:00:13.154103 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:13.154025 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" path="/var/lib/kubelet/pods/f2140e80-7018-4cad-ac47-058943f7e7cc/volumes" Sep 15 13:00:19.294866 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:19.294839 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 13:00:19.295322 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:19.295307 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 13:00:19.751785 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:19.751756 2550 scope.go:117] "RemoveContainer" containerID="a8b433bbe6d396ca1a283b1d3a781e8b508dbe9cfd251e6a2f951c5d93a6e455" Sep 15 13:00:19.760201 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:19.760184 2550 scope.go:117] "RemoveContainer" containerID="35368dd508fe533699c1225882c1b9388f4161f618ad929756a230a55446c4c8" Sep 15 13:00:24.920630 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:24.920597 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst"] Sep 15 13:00:24.925045 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:24.925018 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f0171b-29824620-rgrst"] Sep 15 13:00:24.948987 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:24.948959 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9"] Sep 15 13:00:24.949208 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:24.949165 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" podUID="a3f76539-5719-467b-b2be-47f030ca9160" containerName="payload-pre-processing" containerID="cri-o://4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29" gracePeriod=30 Sep 15 13:00:25.109739 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.109710 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd"] Sep 15 13:00:25.114337 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.114312 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0171b-x58gd"] Sep 15 13:00:25.154707 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.154667 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6507dd30-6419-4eac-8501-4789264c9c88" path="/var/lib/kubelet/pods/6507dd30-6419-4eac-8501-4789264c9c88/volumes" Sep 15 13:00:25.154985 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.154972 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba54e8e-96a2-4b9e-a773-eccd2c761195" path="/var/lib/kubelet/pods/bba54e8e-96a2-4b9e-a773-eccd2c761195/volumes" Sep 15 13:00:25.216212 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.216185 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 13:00:25.314735 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.314639 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f76539-5719-467b-b2be-47f030ca9160-plugins-config-volume\") pod \"a3f76539-5719-467b-b2be-47f030ca9160\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " Sep 15 13:00:25.314899 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.314787 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/a3f76539-5719-467b-b2be-47f030ca9160-kube-api-access-bhzp2\") pod \"a3f76539-5719-467b-b2be-47f030ca9160\" (UID: \"a3f76539-5719-467b-b2be-47f030ca9160\") " Sep 15 13:00:25.315018 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.314996 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f76539-5719-467b-b2be-47f030ca9160-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3f76539-5719-467b-b2be-47f030ca9160" (UID: "a3f76539-5719-467b-b2be-47f030ca9160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:00:25.315194 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.315172 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3f76539-5719-467b-b2be-47f030ca9160-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:25.316819 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.316783 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f76539-5719-467b-b2be-47f030ca9160-kube-api-access-bhzp2" (OuterVolumeSpecName: "kube-api-access-bhzp2") pod "a3f76539-5719-467b-b2be-47f030ca9160" (UID: "a3f76539-5719-467b-b2be-47f030ca9160"). InnerVolumeSpecName "kube-api-access-bhzp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:00:25.416487 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.416450 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzp2\" (UniqueName: \"kubernetes.io/projected/a3f76539-5719-467b-b2be-47f030ca9160-kube-api-access-bhzp2\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:00:25.967807 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.967773 2550 generic.go:356] "Generic (PLEG): container finished" podID="a3f76539-5719-467b-b2be-47f030ca9160" containerID="4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29" exitCode=0 Sep 15 13:00:25.968204 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.967815 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" Sep 15 13:00:25.968204 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.967855 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" event={"ID":"a3f76539-5719-467b-b2be-47f030ca9160","Type":"ContainerDied","Data":"4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29"} Sep 15 13:00:25.968204 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.967904 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9" event={"ID":"a3f76539-5719-467b-b2be-47f030ca9160","Type":"ContainerDied","Data":"567f0b151e82574b6353852636e3cb2b7534941899a86e459d876b5580f7299c"} Sep 15 13:00:25.968204 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.967927 2550 scope.go:117] "RemoveContainer" containerID="4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29" Sep 15 13:00:25.976825 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.976806 2550 scope.go:117] "RemoveContainer" containerID="4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29" Sep 15 13:00:25.977100 ip-10-0-137-242 kubenswrapper[2550]: E0915 13:00:25.977081 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29\": container with ID starting with 4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29 not found: ID does not exist" containerID="4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29" Sep 15 13:00:25.977143 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.977109 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29"} err="failed to get container status \"4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29\": rpc error: code = NotFound desc = could not find container \"4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29\": container with ID starting with 4c976b82df286460999964c54a0378126e68e02a552e764e2b5887ce53e03c29 not found: ID does not exist" Sep 15 13:00:25.987420 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.987395 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9"] Sep 15 13:00:25.992950 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:25.992930 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0171b-555665f98b-4tct9"] Sep 15 13:00:27.154496 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:00:27.154459 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3f76539-5719-467b-b2be-47f030ca9160" path="/var/lib/kubelet/pods/a3f76539-5719-467b-b2be-47f030ca9160/volumes" Sep 15 13:01:14.071649 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.071565 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8"] Sep 15 13:01:14.072130 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.071790 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" containerID="cri-o://236aa0b470c7285cc6168492e98c9ab926205c92fc285f187656f5a65948b65e" gracePeriod=30 Sep 15 13:01:14.477174 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477088 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb"] Sep 15 13:01:14.477578 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477563 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3f76539-5719-467b-b2be-47f030ca9160" containerName="payload-pre-processing" Sep 15 13:01:14.477627 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477582 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3f76539-5719-467b-b2be-47f030ca9160" containerName="payload-pre-processing" Sep 15 13:01:14.477627 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477595 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" Sep 15 13:01:14.477627 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477601 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" Sep 15 13:01:14.477627 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477617 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6507dd30-6419-4eac-8501-4789264c9c88" containerName="revoke-keys" Sep 15 13:01:14.477627 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477624 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6507dd30-6419-4eac-8501-4789264c9c88" containerName="revoke-keys" Sep 15 13:01:14.477800 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477696 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6507dd30-6419-4eac-8501-4789264c9c88" containerName="revoke-keys" Sep 15 13:01:14.477800 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477705 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" Sep 15 13:01:14.477800 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477712 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3f76539-5719-467b-b2be-47f030ca9160" containerName="payload-pre-processing" Sep 15 13:01:14.477800 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.477720 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" Sep 15 13:01:14.480994 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.480979 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" Sep 15 13:01:14.483314 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.483295 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 13:01:14.483314 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.483307 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 13:01:14.489128 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.488663 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb"] Sep 15 13:01:14.624253 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.624214 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0afdec7-6d2a-4f61-8534-48eff863af27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb\" (UID: \"d0afdec7-6d2a-4f61-8534-48eff863af27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" Sep 15 13:01:14.725437 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.725403 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0afdec7-6d2a-4f61-8534-48eff863af27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb\" (UID: \"d0afdec7-6d2a-4f61-8534-48eff863af27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" Sep 15 13:01:14.726037 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.726017 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0afdec7-6d2a-4f61-8534-48eff863af27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb\" (UID: \"d0afdec7-6d2a-4f61-8534-48eff863af27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" Sep 15 13:01:14.793571 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:14.793539 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" Sep 15 13:01:15.066850 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:15.066819 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb"] Sep 15 13:01:15.068448 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:01:15.068416 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0afdec7_6d2a_4f61_8534_48eff863af27.slice/crio-721cae08c7903a8887f7df63026101d6a9e70843d6871b258dbfb6b984a1b73d WatchSource:0}: Error finding container 721cae08c7903a8887f7df63026101d6a9e70843d6871b258dbfb6b984a1b73d: Status 404 returned error can't find the container with id 721cae08c7903a8887f7df63026101d6a9e70843d6871b258dbfb6b984a1b73d Sep 15 13:01:15.241755 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:15.241726 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" event={"ID":"d0afdec7-6d2a-4f61-8534-48eff863af27","Type":"ContainerStarted","Data":"721cae08c7903a8887f7df63026101d6a9e70843d6871b258dbfb6b984a1b73d"} Sep 15 13:01:16.248661 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:16.248621 2550 generic.go:356] "Generic (PLEG): container finished" podID="d0afdec7-6d2a-4f61-8534-48eff863af27" containerID="b6b875a2e0791ff6ce6e9aa0d5a1fd2b9a34c5957cf0f851afc925b725424d65" exitCode=0 Sep 15 13:01:16.249052 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:16.248674 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" event={"ID":"d0afdec7-6d2a-4f61-8534-48eff863af27","Type":"ContainerDied","Data":"b6b875a2e0791ff6ce6e9aa0d5a1fd2b9a34c5957cf0f851afc925b725424d65"} Sep 15 13:01:17.398197 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:17.398173 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" Sep 15 13:01:17.452710 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:17.452668 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0afdec7-6d2a-4f61-8534-48eff863af27-maas-api-service-ca\") pod \"d0afdec7-6d2a-4f61-8534-48eff863af27\" (UID: \"d0afdec7-6d2a-4f61-8534-48eff863af27\") " Sep 15 13:01:17.453062 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:17.453035 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0afdec7-6d2a-4f61-8534-48eff863af27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0afdec7-6d2a-4f61-8534-48eff863af27" (UID: "d0afdec7-6d2a-4f61-8534-48eff863af27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:01:17.553595 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:17.553492 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0afdec7-6d2a-4f61-8534-48eff863af27-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:18.261595 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:18.261567 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" Sep 15 13:01:18.261777 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:18.261568 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb" event={"ID":"d0afdec7-6d2a-4f61-8534-48eff863af27","Type":"ContainerDied","Data":"721cae08c7903a8887f7df63026101d6a9e70843d6871b258dbfb6b984a1b73d"} Sep 15 13:01:18.261777 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:18.261681 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721cae08c7903a8887f7df63026101d6a9e70843d6871b258dbfb6b984a1b73d" Sep 15 13:01:19.271099 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.271064 2550 generic.go:356] "Generic (PLEG): container finished" podID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerID="236aa0b470c7285cc6168492e98c9ab926205c92fc285f187656f5a65948b65e" exitCode=0 Sep 15 13:01:19.271533 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.271136 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" event={"ID":"6b316189-29eb-4bc7-95eb-d0ea6325c2a0","Type":"ContainerDied","Data":"236aa0b470c7285cc6168492e98c9ab926205c92fc285f187656f5a65948b65e"} Sep 15 13:01:19.271533 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.271219 2550 scope.go:117] "RemoveContainer" containerID="083f04de6a189d2b595c28dfa3c141fbe87d9766c7c4c8607dcf3cbb04db31f6" Sep 15 13:01:19.332138 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.332078 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 13:01:19.367081 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367047 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-data\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367273 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367115 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtxr6\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-kube-api-access-vtxr6\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367273 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367166 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-socket\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367273 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367205 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istiod-ca-cert\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367273 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367251 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-certs\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367508 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367283 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-token\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367508 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367360 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-podinfo\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367508 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367400 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:19.367508 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367425 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-credential-socket\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367508 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367458 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-envoy\") pod \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\" (UID: \"6b316189-29eb-4bc7-95eb-d0ea6325c2a0\") " Sep 15 13:01:19.367771 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367662 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:19.367771 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367709 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:19.367879 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367822 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.367879 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367841 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.367879 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.367855 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.368069 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.368032 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:01:19.368393 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.368353 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:19.369888 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.369856 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 13:01:19.370095 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.370065 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:19.370194 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.370102 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:01:19.370194 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.370115 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-kube-api-access-vtxr6" (OuterVolumeSpecName: "kube-api-access-vtxr6") pod "6b316189-29eb-4bc7-95eb-d0ea6325c2a0" (UID: "6b316189-29eb-4bc7-95eb-d0ea6325c2a0"). InnerVolumeSpecName "kube-api-access-vtxr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:01:19.469243 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.469201 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.469243 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.469237 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.469243 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.469250 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.469529 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.469261 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.469529 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.469272 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:19.469529 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:19.469284 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtxr6\" (UniqueName: \"kubernetes.io/projected/6b316189-29eb-4bc7-95eb-d0ea6325c2a0-kube-api-access-vtxr6\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:20.278284 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:20.278251 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" Sep 15 13:01:20.278720 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:20.278294 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8" event={"ID":"6b316189-29eb-4bc7-95eb-d0ea6325c2a0","Type":"ContainerDied","Data":"b13e74d72033497c882e151d35f5c1cd70580efa7956b4f8b05fe884c1901362"} Sep 15 13:01:20.278720 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:20.278357 2550 scope.go:117] "RemoveContainer" containerID="236aa0b470c7285cc6168492e98c9ab926205c92fc285f187656f5a65948b65e" Sep 15 13:01:20.299437 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:20.299400 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8"] Sep 15 13:01:20.305026 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:20.304996 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0171b-openshift-default-5f64c7b96d-kg4k8"] Sep 15 13:01:21.153928 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:21.153891 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" path="/var/lib/kubelet/pods/6b316189-29eb-4bc7-95eb-d0ea6325c2a0/volumes" Sep 15 13:01:24.925009 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:24.924959 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj"] Sep 15 13:01:24.927231 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:24.927207 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cd72a9-29824620-9lzkj"] Sep 15 13:01:25.115327 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:25.115292 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb"] Sep 15 13:01:25.117546 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:25.117522 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cd72a9-94xwb"] Sep 15 13:01:25.155435 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:25.155398 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cb9325-41d4-4920-b35b-a4b6cf8f0136" path="/var/lib/kubelet/pods/43cb9325-41d4-4920-b35b-a4b6cf8f0136/volumes" Sep 15 13:01:25.155750 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:25.155736 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0afdec7-6d2a-4f61-8534-48eff863af27" path="/var/lib/kubelet/pods/d0afdec7-6d2a-4f61-8534-48eff863af27/volumes" Sep 15 13:01:31.743213 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:31.743175 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w"] Sep 15 13:01:31.744019 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:31.743990 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" containerID="cri-o://2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380" gracePeriod=30 Sep 15 13:01:32.102232 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102150 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j"] Sep 15 13:01:32.102581 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102568 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" Sep 15 13:01:32.102635 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102583 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" Sep 15 13:01:32.102635 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102591 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0afdec7-6d2a-4f61-8534-48eff863af27" containerName="revoke-keys" Sep 15 13:01:32.102635 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102597 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0afdec7-6d2a-4f61-8534-48eff863af27" containerName="revoke-keys" Sep 15 13:01:32.102635 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102620 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" Sep 15 13:01:32.102635 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102625 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" Sep 15 13:01:32.102635 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102631 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" Sep 15 13:01:32.102635 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102636 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2140e80-7018-4cad-ac47-058943f7e7cc" containerName="istio-proxy" Sep 15 13:01:32.102848 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102692 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" Sep 15 13:01:32.102848 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102701 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0afdec7-6d2a-4f61-8534-48eff863af27" containerName="revoke-keys" Sep 15 13:01:32.102848 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.102711 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b316189-29eb-4bc7-95eb-d0ea6325c2a0" containerName="istio-proxy" Sep 15 13:01:32.106050 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.106024 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" Sep 15 13:01:32.108223 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.108198 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 13:01:32.108309 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.108248 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 13:01:32.112668 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.112644 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j"] Sep 15 13:01:32.198961 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.198921 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248520a2-7d69-495c-8f81-462169134cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j\" (UID: \"248520a2-7d69-495c-8f81-462169134cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" Sep 15 13:01:32.300485 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.300449 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248520a2-7d69-495c-8f81-462169134cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j\" (UID: \"248520a2-7d69-495c-8f81-462169134cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" Sep 15 13:01:32.301071 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.301048 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248520a2-7d69-495c-8f81-462169134cab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j\" (UID: \"248520a2-7d69-495c-8f81-462169134cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" Sep 15 13:01:32.417955 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.417869 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" Sep 15 13:01:32.687508 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:32.687483 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j"] Sep 15 13:01:32.689811 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:01:32.689780 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248520a2_7d69_495c_8f81_462169134cab.slice/crio-76e23b8164a0c8dfa62b73974e73c1fcd2bab5950fec7698b711849f3ae32f6e WatchSource:0}: Error finding container 76e23b8164a0c8dfa62b73974e73c1fcd2bab5950fec7698b711849f3ae32f6e: Status 404 returned error can't find the container with id 76e23b8164a0c8dfa62b73974e73c1fcd2bab5950fec7698b711849f3ae32f6e Sep 15 13:01:33.351189 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:33.351159 2550 generic.go:356] "Generic (PLEG): container finished" podID="248520a2-7d69-495c-8f81-462169134cab" containerID="18b2a375006d1f2fba5863d1d24d64f2a86fc16ae07d5c1041ef3b90f60436e7" exitCode=0 Sep 15 13:01:33.351507 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:33.351241 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" event={"ID":"248520a2-7d69-495c-8f81-462169134cab","Type":"ContainerDied","Data":"18b2a375006d1f2fba5863d1d24d64f2a86fc16ae07d5c1041ef3b90f60436e7"} Sep 15 13:01:33.351507 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:33.351299 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" event={"ID":"248520a2-7d69-495c-8f81-462169134cab","Type":"ContainerStarted","Data":"76e23b8164a0c8dfa62b73974e73c1fcd2bab5950fec7698b711849f3ae32f6e"} Sep 15 13:01:34.500085 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:34.500059 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" Sep 15 13:01:34.623732 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:34.623644 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248520a2-7d69-495c-8f81-462169134cab-maas-api-service-ca\") pod \"248520a2-7d69-495c-8f81-462169134cab\" (UID: \"248520a2-7d69-495c-8f81-462169134cab\") " Sep 15 13:01:34.623973 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:34.623950 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248520a2-7d69-495c-8f81-462169134cab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "248520a2-7d69-495c-8f81-462169134cab" (UID: "248520a2-7d69-495c-8f81-462169134cab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:01:34.725282 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:34.725248 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248520a2-7d69-495c-8f81-462169134cab-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:35.364462 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:35.364428 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" Sep 15 13:01:35.364630 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:35.364474 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j" event={"ID":"248520a2-7d69-495c-8f81-462169134cab","Type":"ContainerDied","Data":"76e23b8164a0c8dfa62b73974e73c1fcd2bab5950fec7698b711849f3ae32f6e"} Sep 15 13:01:35.364630 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:35.364526 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76e23b8164a0c8dfa62b73974e73c1fcd2bab5950fec7698b711849f3ae32f6e" Sep 15 13:01:36.994292 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:36.994261 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 13:01:37.149680 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149582 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-data\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149680 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149653 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-socket\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149913 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149682 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9ktf\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-kube-api-access-d9ktf\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149913 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149711 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-envoy\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149913 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149738 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-token\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149913 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149764 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-podinfo\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149913 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149798 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-certs\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149913 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149842 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23ab45e-850b-4206-8cc9-19add3dd35f6-istiod-ca-cert\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.149913 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149872 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-credential-socket\") pod \"a23ab45e-850b-4206-8cc9-19add3dd35f6\" (UID: \"a23ab45e-850b-4206-8cc9-19add3dd35f6\") " Sep 15 13:01:37.150253 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149904 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:37.150253 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.149917 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:37.150253 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.150142 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:37.150253 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.150194 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.150253 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.150211 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.150253 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.150228 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23ab45e-850b-4206-8cc9-19add3dd35f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:01:37.150578 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.150463 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:37.152282 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.152250 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-kube-api-access-d9ktf" (OuterVolumeSpecName: "kube-api-access-d9ktf") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "kube-api-access-d9ktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:01:37.152282 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.152261 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 13:01:37.152468 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.152423 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:37.152468 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.152452 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a23ab45e-850b-4206-8cc9-19add3dd35f6" (UID: "a23ab45e-850b-4206-8cc9-19add3dd35f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:01:37.251232 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.251193 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.251232 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.251227 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23ab45e-850b-4206-8cc9-19add3dd35f6-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.251513 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.251314 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.251513 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.251332 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9ktf\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-kube-api-access-d9ktf\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.251513 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.251355 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.251513 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.251367 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.251513 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.251398 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23ab45e-850b-4206-8cc9-19add3dd35f6-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:37.377187 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.377150 2550 generic.go:356] "Generic (PLEG): container finished" podID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerID="2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380" exitCode=0 Sep 15 13:01:37.377370 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.377199 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" Sep 15 13:01:37.377370 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.377237 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" event={"ID":"a23ab45e-850b-4206-8cc9-19add3dd35f6","Type":"ContainerDied","Data":"2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380"} Sep 15 13:01:37.377370 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.377305 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w" event={"ID":"a23ab45e-850b-4206-8cc9-19add3dd35f6","Type":"ContainerDied","Data":"20a0922c805d6a113f4f092cab016f537053892a147aa2f698aeb1297fe1cf6e"} Sep 15 13:01:37.377370 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.377336 2550 scope.go:117] "RemoveContainer" containerID="2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380" Sep 15 13:01:37.386803 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.386785 2550 scope.go:117] "RemoveContainer" containerID="1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be" Sep 15 13:01:37.393193 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.393164 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w"] Sep 15 13:01:37.396197 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.396178 2550 scope.go:117] "RemoveContainer" containerID="2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380" Sep 15 13:01:37.396524 ip-10-0-137-242 kubenswrapper[2550]: E0915 13:01:37.396502 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380\": container with ID starting with 2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380 not found: ID does not exist" containerID="2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380" Sep 15 13:01:37.396573 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.396534 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380"} err="failed to get container status \"2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380\": rpc error: code = NotFound desc = could not find container \"2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380\": container with ID starting with 2a6d6a1eb38be52a698a2c362d31641c7a3284c745da0b8f041e73055ba76380 not found: ID does not exist" Sep 15 13:01:37.396573 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.396552 2550 scope.go:117] "RemoveContainer" containerID="1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be" Sep 15 13:01:37.396826 ip-10-0-137-242 kubenswrapper[2550]: E0915 13:01:37.396807 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be\": container with ID starting with 1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be not found: ID does not exist" containerID="1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be" Sep 15 13:01:37.396872 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.396831 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be"} err="failed to get container status \"1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be\": rpc error: code = NotFound desc = could not find container \"1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be\": container with ID starting with 1999b721d28ec4f7e65b95d1eb02178b821741519abd239d730a35eba9d264be not found: ID does not exist" Sep 15 13:01:37.400084 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:37.400032 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cd72a9-openshift-default-659dc7c9b4-vg86w"] Sep 15 13:01:39.153827 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:39.153789 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" path="/var/lib/kubelet/pods/a23ab45e-850b-4206-8cc9-19add3dd35f6/volumes" Sep 15 13:01:42.606541 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:42.606508 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss"] Sep 15 13:01:42.608422 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:42.608392 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-29a9e9-29824620-ds9ss"] Sep 15 13:01:42.639746 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:42.639709 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs"] Sep 15 13:01:42.640017 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:42.639990 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" podUID="4e0642b9-7db2-465f-af5a-eadfbb0ce164" containerName="payload-processing" containerID="cri-o://9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831" gracePeriod=30 Sep 15 13:01:42.804277 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:42.804239 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j"] Sep 15 13:01:42.807324 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:42.807295 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-29a9e9-6852j"] Sep 15 13:01:42.904443 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:42.904415 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 13:01:43.005934 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.005897 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0642b9-7db2-465f-af5a-eadfbb0ce164-plugins-config-volume\") pod \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " Sep 15 13:01:43.006113 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.005946 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qqbb\" (UniqueName: \"kubernetes.io/projected/4e0642b9-7db2-465f-af5a-eadfbb0ce164-kube-api-access-7qqbb\") pod \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\" (UID: \"4e0642b9-7db2-465f-af5a-eadfbb0ce164\") " Sep 15 13:01:43.006431 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.006371 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0642b9-7db2-465f-af5a-eadfbb0ce164-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e0642b9-7db2-465f-af5a-eadfbb0ce164" (UID: "4e0642b9-7db2-465f-af5a-eadfbb0ce164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:01:43.008177 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.008149 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0642b9-7db2-465f-af5a-eadfbb0ce164-kube-api-access-7qqbb" (OuterVolumeSpecName: "kube-api-access-7qqbb") pod "4e0642b9-7db2-465f-af5a-eadfbb0ce164" (UID: "4e0642b9-7db2-465f-af5a-eadfbb0ce164"). InnerVolumeSpecName "kube-api-access-7qqbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:01:43.106712 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.106674 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqbb\" (UniqueName: \"kubernetes.io/projected/4e0642b9-7db2-465f-af5a-eadfbb0ce164-kube-api-access-7qqbb\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:43.106712 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.106705 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e0642b9-7db2-465f-af5a-eadfbb0ce164-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:43.154417 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.154321 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248520a2-7d69-495c-8f81-462169134cab" path="/var/lib/kubelet/pods/248520a2-7d69-495c-8f81-462169134cab/volumes" Sep 15 13:01:43.154676 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.154662 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9445b8d-c0a8-4331-b658-2f640705c8dd" path="/var/lib/kubelet/pods/e9445b8d-c0a8-4331-b658-2f640705c8dd/volumes" Sep 15 13:01:43.412320 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.412224 2550 generic.go:356] "Generic (PLEG): container finished" podID="4e0642b9-7db2-465f-af5a-eadfbb0ce164" containerID="9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831" exitCode=0 Sep 15 13:01:43.412320 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.412269 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" Sep 15 13:01:43.412557 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.412312 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" event={"ID":"4e0642b9-7db2-465f-af5a-eadfbb0ce164","Type":"ContainerDied","Data":"9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831"} Sep 15 13:01:43.412557 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.412361 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs" event={"ID":"4e0642b9-7db2-465f-af5a-eadfbb0ce164","Type":"ContainerDied","Data":"6ed370221f3469fb84150e04852487db5520b54d8b802217b764fc476e939dbf"} Sep 15 13:01:43.412557 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.412403 2550 scope.go:117] "RemoveContainer" containerID="9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831" Sep 15 13:01:43.421680 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.421660 2550 scope.go:117] "RemoveContainer" containerID="9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831" Sep 15 13:01:43.421997 ip-10-0-137-242 kubenswrapper[2550]: E0915 13:01:43.421971 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831\": container with ID starting with 9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831 not found: ID does not exist" containerID="9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831" Sep 15 13:01:43.422056 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.422008 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831"} err="failed to get container status \"9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831\": rpc error: code = NotFound desc = could not find container \"9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831\": container with ID starting with 9c32942408e6a9547614154c88cdc2bec5284266ebdf5198bab932a073b54831 not found: ID does not exist" Sep 15 13:01:43.429676 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.429646 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs"] Sep 15 13:01:43.433374 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:43.433347 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-29a9e9-779d657f7-hqjhs"] Sep 15 13:01:45.155139 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:45.155100 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0642b9-7db2-465f-af5a-eadfbb0ce164" path="/var/lib/kubelet/pods/4e0642b9-7db2-465f-af5a-eadfbb0ce164/volumes" Sep 15 13:01:49.278733 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.278700 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp"] Sep 15 13:01:49.279187 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.278910 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" containerID="cri-o://afe390bef12cb9e48496d5898ff993d4d28eafcc01e4af09bbee874064c0c499" gracePeriod=30 Sep 15 13:01:49.635477 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635395 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg"] Sep 15 13:01:49.635799 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635786 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" Sep 15 13:01:49.635842 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635800 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" Sep 15 13:01:49.635842 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635809 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248520a2-7d69-495c-8f81-462169134cab" containerName="revoke-keys" Sep 15 13:01:49.635842 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635814 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="248520a2-7d69-495c-8f81-462169134cab" containerName="revoke-keys" Sep 15 13:01:49.635842 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635827 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" Sep 15 13:01:49.635842 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635833 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" Sep 15 13:01:49.635997 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635852 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0642b9-7db2-465f-af5a-eadfbb0ce164" containerName="payload-processing" Sep 15 13:01:49.635997 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635857 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0642b9-7db2-465f-af5a-eadfbb0ce164" containerName="payload-processing" Sep 15 13:01:49.635997 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635914 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" Sep 15 13:01:49.635997 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635922 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23ab45e-850b-4206-8cc9-19add3dd35f6" containerName="istio-proxy" Sep 15 13:01:49.635997 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635929 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="248520a2-7d69-495c-8f81-462169134cab" containerName="revoke-keys" Sep 15 13:01:49.635997 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.635935 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0642b9-7db2-465f-af5a-eadfbb0ce164" containerName="payload-processing" Sep 15 13:01:49.640452 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.640433 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" Sep 15 13:01:49.642470 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.642440 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 13:01:49.642597 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.642478 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hzm2x\"" Sep 15 13:01:49.646029 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.646002 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg"] Sep 15 13:01:49.773041 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.773000 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730490b-ace7-4dd4-be6c-2cd813b38ec0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg\" (UID: \"5730490b-ace7-4dd4-be6c-2cd813b38ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" Sep 15 13:01:49.873917 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.873867 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730490b-ace7-4dd4-be6c-2cd813b38ec0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg\" (UID: \"5730490b-ace7-4dd4-be6c-2cd813b38ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" Sep 15 13:01:49.874570 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.874549 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730490b-ace7-4dd4-be6c-2cd813b38ec0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg\" (UID: \"5730490b-ace7-4dd4-be6c-2cd813b38ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" Sep 15 13:01:49.953118 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:49.953023 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" Sep 15 13:01:50.222809 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:50.222752 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg"] Sep 15 13:01:50.225239 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:01:50.225205 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5730490b_ace7_4dd4_be6c_2cd813b38ec0.slice/crio-95b8ddf253c6ae30ee4ec6624964bbf8a7eb2fb05e281e1d8b173e1bf286277c WatchSource:0}: Error finding container 95b8ddf253c6ae30ee4ec6624964bbf8a7eb2fb05e281e1d8b173e1bf286277c: Status 404 returned error can't find the container with id 95b8ddf253c6ae30ee4ec6624964bbf8a7eb2fb05e281e1d8b173e1bf286277c Sep 15 13:01:50.454717 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:50.454687 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" event={"ID":"5730490b-ace7-4dd4-be6c-2cd813b38ec0","Type":"ContainerStarted","Data":"95b8ddf253c6ae30ee4ec6624964bbf8a7eb2fb05e281e1d8b173e1bf286277c"} Sep 15 13:01:51.461338 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:51.461298 2550 generic.go:356] "Generic (PLEG): container finished" podID="5730490b-ace7-4dd4-be6c-2cd813b38ec0" containerID="dd9aaa36adacbee32bef9051d63ade43a6843b5927700c55253a99a32d941d99" exitCode=0 Sep 15 13:01:51.461729 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:51.461397 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" event={"ID":"5730490b-ace7-4dd4-be6c-2cd813b38ec0","Type":"ContainerDied","Data":"dd9aaa36adacbee32bef9051d63ade43a6843b5927700c55253a99a32d941d99"} Sep 15 13:01:52.605773 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:52.605744 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" Sep 15 13:01:52.701765 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:52.701727 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730490b-ace7-4dd4-be6c-2cd813b38ec0-maas-api-service-ca\") pod \"5730490b-ace7-4dd4-be6c-2cd813b38ec0\" (UID: \"5730490b-ace7-4dd4-be6c-2cd813b38ec0\") " Sep 15 13:01:52.702102 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:52.702077 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5730490b-ace7-4dd4-be6c-2cd813b38ec0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5730490b-ace7-4dd4-be6c-2cd813b38ec0" (UID: "5730490b-ace7-4dd4-be6c-2cd813b38ec0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:01:52.802785 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:52.802746 2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5730490b-ace7-4dd4-be6c-2cd813b38ec0-maas-api-service-ca\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:53.473942 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:53.473857 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" Sep 15 13:01:53.473942 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:53.473878 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg" event={"ID":"5730490b-ace7-4dd4-be6c-2cd813b38ec0","Type":"ContainerDied","Data":"95b8ddf253c6ae30ee4ec6624964bbf8a7eb2fb05e281e1d8b173e1bf286277c"} Sep 15 13:01:53.474132 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:53.473947 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b8ddf253c6ae30ee4ec6624964bbf8a7eb2fb05e281e1d8b173e1bf286277c" Sep 15 13:01:54.482951 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.482912 2550 generic.go:356] "Generic (PLEG): container finished" podID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerID="afe390bef12cb9e48496d5898ff993d4d28eafcc01e4af09bbee874064c0c499" exitCode=0 Sep 15 13:01:54.483342 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.482958 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" event={"ID":"46c6bf50-16ee-46a7-88ee-8199cf5c3521","Type":"ContainerDied","Data":"afe390bef12cb9e48496d5898ff993d4d28eafcc01e4af09bbee874064c0c499"} Sep 15 13:01:54.483342 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.483036 2550 scope.go:117] "RemoveContainer" containerID="d63d9afd6ee93d0d0ec93d2d91a60707ae561a41b9ce9ffaaafb145397bbc46f" Sep 15 13:01:54.539791 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.539765 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 13:01:54.619230 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619130 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjv56\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-kube-api-access-wjv56\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619230 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619182 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-credential-socket\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619230 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619205 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-envoy\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619230 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619234 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-socket\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619590 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619253 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-podinfo\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619590 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619291 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-data\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619590 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619310 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istiod-ca-cert\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619590 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619366 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-token\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619590 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619421 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-certs\") pod \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\" (UID: \"46c6bf50-16ee-46a7-88ee-8199cf5c3521\") " Sep 15 13:01:54.619590 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619534 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:54.619911 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619644 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:54.619911 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619742 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-data" (OuterVolumeSpecName: "istio-data") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:54.619911 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619812 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.619911 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619823 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:54.619911 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619834 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-credential-socket\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.619911 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.619834 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:01:54.621605 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.621581 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 13:01:54.621717 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.621661 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-kube-api-access-wjv56" (OuterVolumeSpecName: "kube-api-access-wjv56") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "kube-api-access-wjv56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:01:54.621717 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.621680 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 15 13:01:54.621828 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.621812 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-token" (OuterVolumeSpecName: "istio-token") pod "46c6bf50-16ee-46a7-88ee-8199cf5c3521" (UID: "46c6bf50-16ee-46a7-88ee-8199cf5c3521"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:01:54.720820 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.720770 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-token\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.720820 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.720813 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-workload-certs\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.720820 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.720824 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjv56\" (UniqueName: \"kubernetes.io/projected/46c6bf50-16ee-46a7-88ee-8199cf5c3521-kube-api-access-wjv56\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.720820 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.720833 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-envoy\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.720820 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.720841 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-podinfo\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.721139 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.720848 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istio-data\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:54.721139 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:54.720856 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c6bf50-16ee-46a7-88ee-8199cf5c3521-istiod-ca-cert\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:01:55.496957 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:55.496920 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" Sep 15 13:01:55.497361 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:55.496924 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp" event={"ID":"46c6bf50-16ee-46a7-88ee-8199cf5c3521","Type":"ContainerDied","Data":"e2c59e4cbbeb0db4dc31872d4ba367895a44aaef5eb2d90d260fc45bd8beb5b8"} Sep 15 13:01:55.497361 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:55.497077 2550 scope.go:117] "RemoveContainer" containerID="afe390bef12cb9e48496d5898ff993d4d28eafcc01e4af09bbee874064c0c499" Sep 15 13:01:55.513771 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:55.513743 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp"] Sep 15 13:01:55.516724 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:55.516697 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-29a9e9-openshift-default-77b4db5598-fgfwp"] Sep 15 13:01:57.154803 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:01:57.154767 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" path="/var/lib/kubelet/pods/46c6bf50-16ee-46a7-88ee-8199cf5c3521/volumes" Sep 15 13:02:00.142546 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.142513 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2"] Sep 15 13:02:00.143028 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.142723 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" podUID="12f7998d-e46a-4fe8-a744-88818dd86b62" containerName="payload-processing" containerID="cri-o://520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242" gracePeriod=30 Sep 15 13:02:00.305555 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.305522 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg"] Sep 15 13:02:00.308291 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.308266 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-17f8f8-rwttg"] Sep 15 13:02:00.397539 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.397469 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 13:02:00.469061 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.469024 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f7998d-e46a-4fe8-a744-88818dd86b62-plugins-config-volume\") pod \"12f7998d-e46a-4fe8-a744-88818dd86b62\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " Sep 15 13:02:00.469343 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.469087 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99fh8\" (UniqueName: \"kubernetes.io/projected/12f7998d-e46a-4fe8-a744-88818dd86b62-kube-api-access-99fh8\") pod \"12f7998d-e46a-4fe8-a744-88818dd86b62\" (UID: \"12f7998d-e46a-4fe8-a744-88818dd86b62\") " Sep 15 13:02:00.469524 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.469503 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f7998d-e46a-4fe8-a744-88818dd86b62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12f7998d-e46a-4fe8-a744-88818dd86b62" (UID: "12f7998d-e46a-4fe8-a744-88818dd86b62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 13:02:00.471540 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.471516 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f7998d-e46a-4fe8-a744-88818dd86b62-kube-api-access-99fh8" (OuterVolumeSpecName: "kube-api-access-99fh8") pod "12f7998d-e46a-4fe8-a744-88818dd86b62" (UID: "12f7998d-e46a-4fe8-a744-88818dd86b62"). InnerVolumeSpecName "kube-api-access-99fh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 13:02:00.527690 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.527657 2550 generic.go:356] "Generic (PLEG): container finished" podID="12f7998d-e46a-4fe8-a744-88818dd86b62" containerID="520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242" exitCode=0 Sep 15 13:02:00.527859 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.527701 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" Sep 15 13:02:00.527859 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.527744 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" event={"ID":"12f7998d-e46a-4fe8-a744-88818dd86b62","Type":"ContainerDied","Data":"520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242"} Sep 15 13:02:00.527859 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.527803 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2" event={"ID":"12f7998d-e46a-4fe8-a744-88818dd86b62","Type":"ContainerDied","Data":"07559b27f2cdc919288e1e585df66858750e3ac129263273455ccb6240931144"} Sep 15 13:02:00.527859 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.527830 2550 scope.go:117] "RemoveContainer" containerID="520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242" Sep 15 13:02:00.544307 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.544287 2550 scope.go:117] "RemoveContainer" containerID="520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242" Sep 15 13:02:00.544644 ip-10-0-137-242 kubenswrapper[2550]: E0915 13:02:00.544622 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242\": container with ID starting with 520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242 not found: ID does not exist" containerID="520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242" Sep 15 13:02:00.544719 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.544652 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242"} err="failed to get container status \"520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242\": rpc error: code = NotFound desc = could not find container \"520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242\": container with ID starting with 520db0e74748383c9fa1a7e834c070cafa87a48bcb135ed5c5f80e2a7e201242 not found: ID does not exist" Sep 15 13:02:00.551709 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.551684 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2"] Sep 15 13:02:00.553351 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.553330 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17f8f8-69d9f6798f-f98j2"] Sep 15 13:02:00.570719 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.570694 2550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f7998d-e46a-4fe8-a744-88818dd86b62-plugins-config-volume\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:02:00.570825 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:00.570722 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99fh8\" (UniqueName: \"kubernetes.io/projected/12f7998d-e46a-4fe8-a744-88818dd86b62-kube-api-access-99fh8\") on node \"ip-10-0-137-242.ec2.internal\" DevicePath \"\"" Sep 15 13:02:01.153571 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:01.153536 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f7998d-e46a-4fe8-a744-88818dd86b62" path="/var/lib/kubelet/pods/12f7998d-e46a-4fe8-a744-88818dd86b62/volumes" Sep 15 13:02:01.153937 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:02:01.153865 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5730490b-ace7-4dd4-be6c-2cd813b38ec0" path="/var/lib/kubelet/pods/5730490b-ace7-4dd4-be6c-2cd813b38ec0/volumes" Sep 15 13:05:19.329160 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:05:19.329131 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 13:05:19.330400 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:05:19.330359 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 13:06:19.859602 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:06:19.859571 2550 scope.go:117] "RemoveContainer" containerID="ef14277feebfdbf2036c4352b94b97d2ec2dc675f3fd2128d731e212cd86822f" Sep 15 13:06:19.867899 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:06:19.867881 2550 scope.go:117] "RemoveContainer" containerID="49b1aafa78729cf0e7f35979acc2d20fe0c93e3f2d86ff3f8d8191e1175fb0b1" Sep 15 13:06:19.875565 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:06:19.875545 2550 scope.go:117] "RemoveContainer" containerID="9e2bef73d6e491aad6a6eb97dc562e85160dfcdf4e557aef80a161f063302345" Sep 15 13:06:19.884694 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:06:19.884676 2550 scope.go:117] "RemoveContainer" containerID="ac7749b274e8b843d517128a7e2d35d7d05bcbbff30acc6f2f84cb78751f2c87" Sep 15 13:06:19.891950 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:06:19.891927 2550 scope.go:117] "RemoveContainer" containerID="adb947942bc80180227615ea0642f9450f20a48851db9c3f8e18a4092cd9490d" Sep 15 13:07:19.923678 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:07:19.923600 2550 scope.go:117] "RemoveContainer" containerID="b6b875a2e0791ff6ce6e9aa0d5a1fd2b9a34c5957cf0f851afc925b725424d65" Sep 15 13:08:19.944072 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:19.944039 2550 scope.go:117] "RemoveContainer" containerID="18b2a375006d1f2fba5863d1d24d64f2a86fc16ae07d5c1041ef3b90f60436e7" Sep 15 13:08:19.952420 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:19.952401 2550 scope.go:117] "RemoveContainer" containerID="dd9aaa36adacbee32bef9051d63ade43a6843b5927700c55253a99a32d941d99" Sep 15 13:08:30.986179 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:30.986145 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-z9nkx_c499b483-f74c-4e4b-84e9-e66a2b0941d8/server/0.log" Sep 15 13:08:31.108045 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:31.108006 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8gdgd_7bf764af-d105-4093-86ca-4613af06e0a3/manager/0.log" Sep 15 13:08:31.464208 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:31.464180 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-kc886_b9a6e984-a024-4371-be93-cf69c0653165/copy-manifests/0.log" Sep 15 13:08:31.470911 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:31.470881 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-kc886_b9a6e984-a024-4371-be93-cf69c0653165/manager/0.log" Sep 15 13:08:34.096354 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:34.096322 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ccldkt_0db7cb94-84c4-444c-bdbf-58a3967843d1/istio-proxy/0.log" Sep 15 13:08:34.218973 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:34.218938 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nwd8x_f6aba3fb-247c-4816-8b0a-c5f2c35c06cf/discovery/0.log" Sep 15 13:08:34.792581 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:34.792548 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-v9pnn_107e94e7-82fd-4fda-a35e-28b1d35c5b2b/payload-processing/0.log" Sep 15 13:08:34.905493 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:34.905459 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d9dd5889b-hk9s6_87e210b6-091d-42a9-8d83-403afff524e7/router/0.log" Sep 15 13:08:35.499125 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:35.499094 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-857wl_f15029f5-a5c3-475a-97ab-eacfe278f40c/main/0.log" Sep 15 13:08:35.612971 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:35.612942 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-v8tcv_a28e55bc-4437-4be7-a364-7afa0dbd9067/main/0.log" Sep 15 13:08:44.931174 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:44.931093 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-px8td_ea8d9432-e366-40c2-81fd-b478d05a1df7/global-pull-secret-syncer/0.log" Sep 15 13:08:45.041659 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:45.041630 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qgfwl_09c54c4c-17f5-47a7-8d09-9f185f561093/konnectivity-agent/0.log" Sep 15 13:08:45.089885 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:45.089854 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-242.ec2.internal_f58eb2157143886588c630e14dfef629/haproxy/0.log" Sep 15 13:08:51.959967 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:51.959935 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d96fad9-6b46-462f-81f9-4729c93ed38c/alertmanager/0.log" Sep 15 13:08:51.981046 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:51.981020 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d96fad9-6b46-462f-81f9-4729c93ed38c/config-reloader/0.log" Sep 15 13:08:52.001488 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.001463 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d96fad9-6b46-462f-81f9-4729c93ed38c/kube-rbac-proxy-web/0.log" Sep 15 13:08:52.024439 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.024418 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d96fad9-6b46-462f-81f9-4729c93ed38c/kube-rbac-proxy/0.log" Sep 15 13:08:52.045196 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.045171 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d96fad9-6b46-462f-81f9-4729c93ed38c/kube-rbac-proxy-metric/0.log" Sep 15 13:08:52.068395 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.068356 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d96fad9-6b46-462f-81f9-4729c93ed38c/prom-label-proxy/0.log" Sep 15 13:08:52.095229 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.095206 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7d96fad9-6b46-462f-81f9-4729c93ed38c/init-config-reloader/0.log" Sep 15 13:08:52.146150 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.146124 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-wbvdk_2fbef136-c8c8-4f1e-8226-60ee42734392/cluster-monitoring-operator/0.log" Sep 15 13:08:52.273273 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.273241 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-4hqf9_3cc89b7f-ef8a-496d-9d6f-296856a2a966/monitoring-plugin/0.log" Sep 15 13:08:52.395740 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.395714 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bmwfb_7ca9ce81-ca99-41f1-b380-f621b12b6604/node-exporter/0.log" Sep 15 13:08:52.422546 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.422523 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bmwfb_7ca9ce81-ca99-41f1-b380-f621b12b6604/kube-rbac-proxy/0.log" Sep 15 13:08:52.457813 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.457789 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bmwfb_7ca9ce81-ca99-41f1-b380-f621b12b6604/init-textfile/0.log" Sep 15 13:08:52.842577 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.842548 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8tp4p_bfdc104f-ef60-4779-9dae-b9caf4682c6e/prometheus-operator/0.log" Sep 15 13:08:52.865566 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.865544 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8tp4p_bfdc104f-ef60-4779-9dae-b9caf4682c6e/kube-rbac-proxy/0.log" Sep 15 13:08:52.928608 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.928580 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d989b5bdd-zm4fd_74925c27-0ed3-402a-b309-9d1a57664f8d/telemeter-client/0.log" Sep 15 13:08:52.954286 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.954260 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d989b5bdd-zm4fd_74925c27-0ed3-402a-b309-9d1a57664f8d/reload/0.log" Sep 15 13:08:52.977413 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:52.977364 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d989b5bdd-zm4fd_74925c27-0ed3-402a-b309-9d1a57664f8d/kube-rbac-proxy/0.log" Sep 15 13:08:54.414110 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:54.414078 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-jv74g_373725c3-1275-4adf-99dc-6a3cf09e620b/networking-console-plugin/0.log" Sep 15 13:08:54.998234 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:54.998201 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/1.log" Sep 15 13:08:55.010466 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.010437 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-7chzc_f41fba11-4f50-4a76-8daa-be90c630605c/console-operator/2.log" Sep 15 13:08:55.454663 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.454588 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw"] Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.454952 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5730490b-ace7-4dd4-be6c-2cd813b38ec0" containerName="revoke-keys" Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.454963 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5730490b-ace7-4dd4-be6c-2cd813b38ec0" containerName="revoke-keys" Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.454974 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f7998d-e46a-4fe8-a744-88818dd86b62" containerName="payload-processing" Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.454980 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f7998d-e46a-4fe8-a744-88818dd86b62" containerName="payload-processing" Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455001 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455006 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455014 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" Sep 15 13:08:55.455027 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455020 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" Sep 15 13:08:55.455276 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455078 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" Sep 15 13:08:55.455276 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455084 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="46c6bf50-16ee-46a7-88ee-8199cf5c3521" containerName="istio-proxy" Sep 15 13:08:55.455276 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455091 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f7998d-e46a-4fe8-a744-88818dd86b62" containerName="payload-processing" Sep 15 13:08:55.455276 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.455099 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5730490b-ace7-4dd4-be6c-2cd813b38ec0" containerName="revoke-keys" Sep 15 13:08:55.458314 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.458297 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.460618 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.460596 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fpbvp\"/\"openshift-service-ca.crt\"" Sep 15 13:08:55.461565 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.461545 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fpbvp\"/\"kube-root-ca.crt\"" Sep 15 13:08:55.461648 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.461566 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fpbvp\"/\"default-dockercfg-kswxq\"" Sep 15 13:08:55.467193 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.467170 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw"] Sep 15 13:08:55.570022 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.569991 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-sys\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.570022 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.570026 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-lib-modules\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.570239 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.570057 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-podres\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.570239 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.570125 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-proc\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.570239 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.570150 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqk9\" (UniqueName: \"kubernetes.io/projected/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-kube-api-access-qtqk9\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.661976 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.661947 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-lzq77_25a3f5bc-8afb-46b8-8ea4-0acf695bfc92/download-server/0.log" Sep 15 13:08:55.670778 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670753 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-sys\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.670883 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670785 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-lib-modules\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.670883 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670814 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-podres\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.670883 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670864 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-proc\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.671039 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670885 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-sys\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.671039 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670890 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqk9\" (UniqueName: \"kubernetes.io/projected/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-kube-api-access-qtqk9\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.671039 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670930 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-lib-modules\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.671039 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.670956 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-proc\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.671039 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.671001 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-podres\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.677893 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.677866 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqk9\" (UniqueName: \"kubernetes.io/projected/04ec0882-8342-4a6c-ba2d-d8b79aa48c74-kube-api-access-qtqk9\") pod \"perf-node-gather-daemonset-8c5pw\" (UID: \"04ec0882-8342-4a6c-ba2d-d8b79aa48c74\") " pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:55.769695 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:55.769661 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:56.043770 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:56.043745 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw"] Sep 15 13:08:56.046005 ip-10-0-137-242 kubenswrapper[2550]: W0915 13:08:56.045965 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04ec0882_8342_4a6c_ba2d_d8b79aa48c74.slice/crio-09fdf777e1fbcf518b8db4800768bc8cf5747cf7b2b3b1cc335b6d1698d01cd3 WatchSource:0}: Error finding container 09fdf777e1fbcf518b8db4800768bc8cf5747cf7b2b3b1cc335b6d1698d01cd3: Status 404 returned error can't find the container with id 09fdf777e1fbcf518b8db4800768bc8cf5747cf7b2b3b1cc335b6d1698d01cd3 Sep 15 13:08:56.339514 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:56.339436 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-q5dcb_93c9c4fb-9015-4d54-83c7-44e32e2fd8af/volume-data-source-validator/0.log" Sep 15 13:08:56.649993 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:56.649904 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" event={"ID":"04ec0882-8342-4a6c-ba2d-d8b79aa48c74","Type":"ContainerStarted","Data":"875aae3a1eed259e7a20beccd1815bef7601b711de591438af41a9c37f9ce66b"} Sep 15 13:08:56.649993 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:56.649957 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" event={"ID":"04ec0882-8342-4a6c-ba2d-d8b79aa48c74","Type":"ContainerStarted","Data":"09fdf777e1fbcf518b8db4800768bc8cf5747cf7b2b3b1cc335b6d1698d01cd3"} Sep 15 13:08:56.650476 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:56.650039 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:08:56.664921 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:56.664876 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" podStartSLOduration=1.664862074 podStartE2EDuration="1.664862074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 13:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 13:08:56.663827606 +0000 UTC m=+2618.018673928" watchObservedRunningTime="2026-09-15 13:08:56.664862074 +0000 UTC m=+2618.019708396" Sep 15 13:08:57.454599 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:57.454562 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtkjn_00f55c5a-30e5-465e-a207-67dd23c5523a/dns/0.log" Sep 15 13:08:57.474278 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:57.474254 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mtkjn_00f55c5a-30e5-465e-a207-67dd23c5523a/kube-rbac-proxy/0.log" Sep 15 13:08:57.566131 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:57.566101 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxrzq_d1c623cb-ff74-44c7-bdd3-6e3b23bf5b55/dns-node-resolver/0.log" Sep 15 13:08:58.189959 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:58.189927 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnvms_259030fd-b0aa-4c90-b2b6-b3a82c8e994c/node-ca/0.log" Sep 15 13:08:59.105084 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:59.105050 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ccldkt_0db7cb94-84c4-444c-bdbf-58a3967843d1/istio-proxy/0.log" Sep 15 13:08:59.964547 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:08:59.964511 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nwd8x_f6aba3fb-247c-4816-8b0a-c5f2c35c06cf/discovery/0.log" Sep 15 13:09:00.111863 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:00.111826 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-v9pnn_107e94e7-82fd-4fda-a35e-28b1d35c5b2b/payload-processing/0.log" Sep 15 13:09:00.135333 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:00.135298 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d9dd5889b-hk9s6_87e210b6-091d-42a9-8d83-403afff524e7/router/0.log" Sep 15 13:09:01.100816 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:01.100783 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ct9mg_b512fd0d-b99f-4f92-83f9-a7ace0c64f65/serve-healthcheck-canary/0.log" Sep 15 13:09:01.736684 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:01.736657 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wwn96_e0f7776b-cd11-4862-9b0c-e638e096aaa5/kube-rbac-proxy/0.log" Sep 15 13:09:01.757111 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:01.757083 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wwn96_e0f7776b-cd11-4862-9b0c-e638e096aaa5/exporter/0.log" Sep 15 13:09:01.777968 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:01.777947 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wwn96_e0f7776b-cd11-4862-9b0c-e638e096aaa5/extractor/0.log" Sep 15 13:09:02.664022 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:02.663995 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fpbvp/perf-node-gather-daemonset-8c5pw" Sep 15 13:09:04.328658 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:04.328636 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-z9nkx_c499b483-f74c-4e4b-84e9-e66a2b0941d8/server/0.log" Sep 15 13:09:04.470598 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:04.470566 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8gdgd_7bf764af-d105-4093-86ca-4613af06e0a3/manager/0.log" Sep 15 13:09:04.661263 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:04.661154 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-kc886_b9a6e984-a024-4371-be93-cf69c0653165/manager/0.log" Sep 15 13:09:04.681402 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:04.681358 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-665cd58c4f-kc886_b9a6e984-a024-4371-be93-cf69c0653165/copy-manifests/0.log" Sep 15 13:09:05.951356 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:05.951273 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ccldkt_0db7cb94-84c4-444c-bdbf-58a3967843d1/istio-proxy/0.log" Sep 15 13:09:06.127723 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:06.127695 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b47d9b998-v8d9j_9082decf-f458-46bc-8520-c53854d47e6f/manager/0.log" Sep 15 13:09:06.812829 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:06.812743 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nwd8x_f6aba3fb-247c-4816-8b0a-c5f2c35c06cf/discovery/0.log" Sep 15 13:09:06.916203 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:06.916174 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-v9pnn_107e94e7-82fd-4fda-a35e-28b1d35c5b2b/payload-processing/0.log" Sep 15 13:09:06.930518 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:06.930490 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d9dd5889b-hk9s6_87e210b6-091d-42a9-8d83-403afff524e7/router/0.log" Sep 15 13:09:08.360719 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:08.360688 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799ccldkt_0db7cb94-84c4-444c-bdbf-58a3967843d1/istio-proxy/0.log" Sep 15 13:09:09.230496 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:09.230456 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nwd8x_f6aba3fb-247c-4816-8b0a-c5f2c35c06cf/discovery/0.log" Sep 15 13:09:09.356074 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:09.356033 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-v9pnn_107e94e7-82fd-4fda-a35e-28b1d35c5b2b/payload-processing/0.log" Sep 15 13:09:09.372936 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:09.372911 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5d9dd5889b-hk9s6_87e210b6-091d-42a9-8d83-403afff524e7/router/0.log" Sep 15 13:09:11.759920 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:11.759828 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n5dv8_18be3ccf-b37c-401c-a169-cc2820e6103f/kube-storage-version-migrator-operator/1.log" Sep 15 13:09:11.761465 ip-10-0-137-242 kubenswrapper[2550]: I0915 13:09:11.761441 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n5dv8_18be3ccf-b37c-401c-a169-cc2820e6103f/kube-storage-version-migrator-operator/0.log"